<html>
<head><meta charset="utf-8"><title>optimizing-transitive-closure · t-compiler/wg-nll · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/index.html">t-compiler/wg-nll</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html">optimizing-transitive-closure</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="126439115"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439115" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439115">(May 11 2018 at 23:20)</a>:</h4>
<p>I spent some time optimizing the borrow-check transitive closure computation. The more optimized version no longer computes the full transitive closure at each point; instead, it only computes the transitive closure for those regions that are going out of scope along an edge. This can probably be made better still, but already it makes quite a difference:</p>
<div class="codehilite"><pre><span></span>Mr-Darcy. cargo run --release -- -a TimelyOpt inputs/clap-rs/app-parser-{{impl}}-add_defaults/ --skip-tuples
    Finished release [optimized] target(s) in 0.07s
     Running `target/release/borrow-check -a TimelyOpt &#39;inputs/clap-rs/app-parser-{{impl}}-add_defaults/&#39; --skip-tuples`
--------------------------------------------------
Directory: inputs/clap-rs/app-parser-{{impl}}-add_defaults/
Time: 63.513s
Mr-Darcy. cargo run --release -- -a Naive inputs/clap-rs/app-parser-{{impl}}-add_defaults/ --skip-tuples
    Finished release [optimized] target(s) in 0.22s
     Running `target/release/borrow-check -a Naive &#39;inputs/clap-rs/app-parser-{{impl}}-add_defaults/&#39; --skip-tuples`
--------------------------------------------------
Directory: inputs/clap-rs/app-parser-{{impl}}-add_defaults/
Time: 131.559s
</pre></div>



<a name="126439122"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439122" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439122">(May 11 2018 at 23:21)</a>:</h4>
<p>I wonder if tables work. Let's find out.</p>
<table>
<thead>
<tr>
<th>version</th>
<th>time</th>
</tr>
</thead>
<tbody>
<tr>
<td>before</td>
<td>131s</td>
</tr>
<tr>
<td>after</td>
<td>63s</td>
</tr>
</tbody>
</table>



<a name="126439124"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439124" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439124">(May 11 2018 at 23:21)</a>:</h4>
<p>obviously still some way to go</p>



<a name="126439278"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439278" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439278">(May 11 2018 at 23:26)</a>:</h4>
<p>though I strongly suspect that <a href="https://github.com/rust-lang-nursery/borrow-check/issues/20" target="_blank" title="https://github.com/rust-lang-nursery/borrow-check/issues/20">https://github.com/rust-lang-nursery/borrow-check/issues/20</a> would help a lot -- <span class="user-mention" data-user-id="116013">@qmx</span>, did you ever do anything in that direction btw?</p>



<a name="126439332"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439332" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439332">(May 11 2018 at 23:27)</a>:</h4>
<p>I gotta figure out the best way to do some profiling of the timely stuff though. It's sort of opaque to me still. Frank McSherry recommended adding <code>count</code> calls at one point to try and observe how many tuples were flying around...</p>



<a name="126439406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439406">(May 11 2018 at 23:29)</a>:</h4>
<p>made one other improvement, and now it's done to 40s</p>



<a name="126439790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439790">(May 11 2018 at 23:39)</a>:</h4>
<p><a href="https://github.com/rust-lang-nursery/borrow-check/pull/23" target="_blank" title="https://github.com/rust-lang-nursery/borrow-check/pull/23">opened a pull request</a>. Not sure who might want to review it =)</p>



<a name="126439851"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439851" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439851">(May 11 2018 at 23:40)</a>:</h4>
<p>That's my weekend project</p>



<a name="126439858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439858">(May 11 2018 at 23:40)</a>:</h4>
<p>ok. I'm stuck in GRU for a few more hours so I may poke at it</p>



<a name="126439866"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439866" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439866">(May 11 2018 at 23:41)</a>:</h4>
<p>or something else, I'm going to first see if I can instrument the code more to get more insights</p>



<a name="126439868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439868">(May 11 2018 at 23:41)</a>:</h4>
<p>I still can't believe it's taking 40s :)</p>



<a name="126439869"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439869" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439869">(May 11 2018 at 23:41)</a>:</h4>
<p>Go for it, I'm studying the basics</p>



<a name="126439876"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439876" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439876">(May 11 2018 at 23:41)</a>:</h4>
<p>how many fricking billions of computer cycles is that :P</p>



<a name="126439927"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126439927" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126439927">(May 11 2018 at 23:42)</a>:</h4>
<p>(good news: I checked, and my new analysis produces precisely the same final result as the old one)</p>



<a name="126440179"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440179" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440179">(May 11 2018 at 23:48)</a>:</h4>
<p>woah, this surprises me, I have to admit:</p>



<a name="126440183"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440183" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440183">(May 11 2018 at 23:48)</a>:</h4>
<div class="codehilite"><pre><span></span>Mr-Darcy. Mr-Darcy. wc -l outlives.facts
  534327 outlives.facts
Mr-Darcy. wc -l region_live_at.facts
  782146 region_live_at.facts
Mr-Darcy. wc -l cfg_edge.facts
   51896 cfg_edge.facts
</pre></div>



<a name="126440187"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440187" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440187">(May 11 2018 at 23:48)</a>:</h4>
<p>well, maybe not the region-live-at, but still, those are big numbers</p>



<a name="126440191"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440191" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440191">(May 11 2018 at 23:48)</a>:</h4>
<p>esp. the first one</p>



<a name="126440199"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440199" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440199">(May 11 2018 at 23:49)</a>:</h4>
<p>and an awful lot of those are bidirectional, just by inspection:</p>



<a name="126440200"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440200" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440200">(May 11 2018 at 23:49)</a>:</h4>
<div class="codehilite"><pre><span></span>&quot;\&#39;_#13627r&quot;    &quot;\&#39;_#7r&quot;    &quot;Mid(bb0[3])&quot;
&quot;\&#39;_#13628r&quot;    &quot;\&#39;_#8r&quot;    &quot;Mid(bb0[3])&quot;
&quot;\&#39;_#7r&quot;    &quot;\&#39;_#13627r&quot;    &quot;Mid(bb0[3])&quot;
&quot;\&#39;_#8r&quot;    &quot;\&#39;_#13628r&quot;    &quot;Mid(bb0[3])&quot;
</pre></div>



<a name="126440265"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440265" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440265">(May 11 2018 at 23:50)</a>:</h4>
<p>huh?</p>



<a name="126440281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440281">(May 11 2018 at 23:51)</a>:</h4>
<p>when we have an invariant lifetime or are just requiring equality, we will generate <code>'a: 'b</code> and <code>'b: 'a</code></p>



<a name="126440282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440282">(May 11 2018 at 23:51)</a>:</h4>
<p>so two edges</p>



<a name="126440288"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440288" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440288">(May 11 2018 at 23:51)</a>:</h4>
<p>I am hypothesizing that this maybe contributes to the remarkable number of outlives facts there (500k)</p>



<a name="126440291"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440291" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440291">(May 11 2018 at 23:51)</a>:</h4>
<p>but i'm not sure if that's really true</p>



<a name="126440292"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440292" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440292">(May 11 2018 at 23:51)</a>:</h4>
<p>as I dig a bit more</p>



<a name="126440335"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440335" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440335">(May 11 2018 at 23:52)</a>:</h4>
<p>mm, actually, it does seem to happena  lot</p>



<a name="126440353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440353">(May 11 2018 at 23:52)</a>:</h4>
<p>e.g. here as well:</p>
<div class="codehilite"><pre><span></span>&quot;\&#39;_#563r&quot;  &quot;\&#39;_#559r&quot;  &quot;Mid(bb240[6])&quot;
&quot;\&#39;_#564r&quot;  &quot;\&#39;_#560r&quot;  &quot;Mid(bb240[6])&quot;
&quot;\&#39;_#565r&quot;  &quot;\&#39;_#561r&quot;  &quot;Mid(bb240[6])&quot;
&quot;\&#39;_#566r&quot;  &quot;\&#39;_#562r&quot;  &quot;Mid(bb240[6])&quot;
&quot;\&#39;_#559r&quot;  &quot;\&#39;_#563r&quot;  &quot;Mid(bb240[6])&quot;
&quot;\&#39;_#560r&quot;  &quot;\&#39;_#564r&quot;  &quot;Mid(bb240[6])&quot;
&quot;\&#39;_#561r&quot;  &quot;\&#39;_#565r&quot;  &quot;Mid(bb240[6])&quot;
&quot;\&#39;_#562r&quot;  &quot;\&#39;_#566r&quot;  &quot;Mid(bb240[6])&quot;
</pre></div>



<a name="126440355"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440355" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440355">(May 11 2018 at 23:52)</a>:</h4>
<p>would it make sense to add metadata to an edge? like marking it as a invariant</p>



<a name="126440379"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440379" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440379">(May 11 2018 at 23:53)</a>:</h4>
<p>or even better, do we even care that things are invariant or not?</p>



<a name="126440448"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440448" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440448">(May 11 2018 at 23:55)</a>:</h4>
<p>I'm not sure yet what makes sense, I just think it's interesting.</p>



<a name="126440455"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440455" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440455">(May 11 2018 at 23:55)</a>:</h4>
<p>you could imagine that adding a new relation -- like subset -- just for equality</p>



<a name="126440456"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440456" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440456">(May 11 2018 at 23:56)</a>:</h4>
<p>but it would make everyhting more complex</p>



<a name="126440497"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440497" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440497">(May 11 2018 at 23:56)</a>:</h4>
<p>I'd prefer not to</p>



<a name="126440498"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126440498" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126440498">(May 11 2018 at 23:56)</a>:</h4>
<p>but still might be something we can exploit somehow</p>



<a name="126441172"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126441172" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126441172">(May 12 2018 at 00:18)</a>:</h4>
<p>oh, hmm. I see part of the problem is the hack I did around the return type, which causes outlives relations to be added at <em>all</em> points</p>



<a name="126442152"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126442152" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126442152">(May 12 2018 at 00:54)</a>:</h4>
<p><a href="https://github.com/rust-lang-nursery/borrow-check/issues/24" target="_blank" title="https://github.com/rust-lang-nursery/borrow-check/issues/24">https://github.com/rust-lang-nursery/borrow-check/issues/24</a></p>



<a name="126456458"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126456458" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126456458">(May 12 2018 at 09:59)</a>:</h4>
<p>Here I is! I'm heading over to the repo to try and page in the Datalog and see if I can say something intelligent.  Immediate (superficial) response: if there are equivalence classes of nodes all mutually connected by bi-directional edges, then coalescing them down could help a lot. The SCC logic does this for transitive closure, but it may need to be something different here (not up to speed enough to be sure).</p>



<a name="126457043"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126457043" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126457043">(May 12 2018 at 10:18)</a>:</h4>
<p>The times you are seeing do seem quite large for the number of tuples you are producing as output. By comparison, using DD to do a memory aliasing computation on the linux kernel produces ~1B facts and takes about 200s; a dataflow (null pointer flow) computation on httpd produces 10M output facts and takes about 10s.</p>
<p>I've got the code and am starting to work through it, but one thing that stands out is a large number of "cyclic joins", which are triangle-like joins of the form:</p>
<div class="codehilite"><pre><span></span>Result(x,y,z) &lt;- A(x,y), B(y,z), C(x,z)
</pre></div>


<p>These can be painful because the <code>A(x,y), B(y,z)</code> join may produce many intermediate results, which may then be substantially reduced by <code>C(x,z)</code>. This may be fundamental, but there are some clever ways to execute these joins (e.g. in <a href="https://github.com/frankmcsherry/differential-dataflow/blob/master/examples/ngo.rs" target="_blank" title="https://github.com/frankmcsherry/differential-dataflow/blob/master/examples/ngo.rs">this example of triangle counting using differential dataflow</a>). </p>
<p>I'll take a look and see if re-wiring these joins to use the improved join patterns helps. One way to "check" is to take the final results for the relations and check out the cardinalities. If for example <code>y</code> has a high degree in <code>A</code> and <code>B</code> then <code>A(x,y),B(y,z)</code> will probably be massive, perhaps much larger than <code>C</code>, and the improved algorithms will likely help.</p>



<a name="126457241"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126457241" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126457241">(May 12 2018 at 10:25)</a>:</h4>
<p>Another optimization that the triangle example above points out, which could shave constant factors but not asymptotic terms, is to use the "arrangement" infrastructure to arrange each (collection, key) pair at most once. </p>
<p>For example, there is a lot of </p>
<div class="codehilite"><pre><span></span><span class="n">some_other_collection</span><span class="p">.</span><span class="n">semijoin</span><span class="p">(</span><span class="o">&amp;</span><span class="n">region_live_at</span><span class="p">);</span><span class="w"></span>
</pre></div>


<p>in the code, and each invocation will form a new indexed form of the contents of <code>region_live_at</code>. Instead, you can write</p>
<div class="codehilite"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">region_live_at_by_self</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">region_live_at</span><span class="p">.</span><span class="n">arrange_by_self</span><span class="p">();</span><span class="w"></span>
</pre></div>


<p>which gives you an arranged stream, one that you can use with <code>join_core</code> to effect a semijoin without rebuilding a <code>region_live_at</code> index for each use:</p>
<div class="codehilite"><pre><span></span><span class="n">some_other_collection</span><span class="p">.</span><span class="n">join_core</span><span class="p">(</span><span class="o">&amp;</span><span class="n">region_live_at_by_self</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="n">key</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="n">val</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="p">()</span><span class="o">|</span><span class="w"> </span><span class="nb">Some</span><span class="p">((</span><span class="n">key</span><span class="p">,</span><span class="w"> </span><span class="n">val</span><span class="p">)));</span><span class="w"></span>
</pre></div>


<p>This can reduce the memory and compute footprint by factors that depend on the amount of re-use of these relations. This will not take anything from 40s to 1s, though.</p>



<a name="126458235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126458235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126458235">(May 12 2018 at 11:03)</a>:</h4>
<p>Another random (possibly out-dated) observation: In the computation of <code>subset</code> there is the rule</p>
<div class="codehilite"><pre><span></span>    // subset(R1, R3, P) :-
    //   subset(R1, R2, P),
    //   subset(R2, R3, P).
</pre></div>


<p>These sorts of rules are "quadratic" and often terrifying. The problem is that when we show up with 7.5M facts in <code>subset</code> (its final size) then there is a potential massive explosion of facts here, even if most of them end up being redundant.</p>
<p>In a transitive closure computation, we would likely instead write</p>
<div class="codehilite"><pre><span></span>    // subset(R1, R3, P) :-
    //   subset(R1, R2, P),
    //   outlives(R2, R3, P).
</pre></div>


<p>which grows <code>subset</code> more slowly (over more rounds) but also doesn't experience a massive (potentially quadratic) derivation spike in any one round. If I make that change with the <code>Naive</code> strategy the time to derive <code>subset</code> drops from 40s down to 15s.</p>
<p>IMPORTANTLY: This change also produces the wrong answer, because you have other <code>subset</code> facts produced by crawling along the <code>cfg_edge</code> relation, so it isn't always applicable, but I thought I'd point it out anyhow as an example of how some restructuring can mitigate some of the performance pain.</p>



<a name="126458521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126458521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126458521">(May 12 2018 at 11:14)</a>:</h4>
<p>Another random observation: in the <code>Naive</code> execution, although <code>subset</code> produces 7.5M distinct derived results, there are actually 59,275,254 derived facts over 1,360 rounds. This implies a certain amount of redundancy, which might be legit but is also starting to explain where the 40s are coming from. :D</p>
<p>Off to look into the <code>TimelyOpt</code> variant and see if I can grok that.</p>



<a name="126458523"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126458523" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126458523">(May 12 2018 at 11:14)</a>:</h4>
<p>IIRC this subset rule was where most of the time came from in the Naive computation (eg removing it dropped from 140s to 9s)</p>



<a name="126458575"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126458575" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126458575">(May 12 2018 at 11:16)</a>:</h4>
<p>(the results were then very wrong of course ;)</p>



<a name="126458701"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126458701" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126458701">(May 12 2018 at 11:21)</a>:</h4>
<p>Looking at the new version of the logic, the first question that comes up (applies to the old version too) is whether it is important to produce all of <code>subset</code> and then determine <code>requires</code>, or whether the derivation of <code>requires</code> can drive the derivation of <code>subset</code>. I.e. it seems we only need <code>subset</code> here:</p>
<div class="codehilite"><pre><span></span>    // requires(R2, B, P) :-
    //   requires(R1, B, P),
    //   subset(R1, R2, P).
</pre></div>


<p>which might mean we can co-derive <code>subset</code> and <code>requires</code> together, restricting the derivation of <code>subset</code> to those <code>(R1, P)</code> present in <code>requires</code>.</p>



<a name="126458809"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126458809" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126458809">(May 12 2018 at 11:25)</a>:</h4>
<p>Similarly, but perhaps less complicatedly, can we fuse the <code>subset</code> logic into the derivation of <code>requires</code>?</p>
<p>In the <code>Naive</code> derivation, we have a bunch of rules like </p>
<div class="codehilite"><pre><span></span>    // subset(R1, R2, P) :- outlives(R1, R2, P).

    // subset(R1, R3, P) :-
    //   subset(R1, R2, P),
    //   subset(R2, R3, P).

    // subset(R1, R2, Q) :-
    // subset(R1, R2, P),
    //   cfg_edge(P, Q),
    //   (region_live_at(R1, Q); universal_region(R1)),
    //   (region_live_at(R2, Q); universal_region(R2)).
</pre></div>


<p>and it seems possible that we could replace </p>
<div class="codehilite"><pre><span></span>     // requires(R2, B, P) :-
     //   requires(R1, B, P),
     //   subset(R1, R2, P).
</pre></div>


<p>with flavors of the above rules and avoid deriving <code>subset</code> at all.</p>



<a name="126459236"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126459236" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126459236">(May 12 2018 at 11:41)</a>:</h4>
<p>Niko asked elsewhere</p>
<blockquote>
<p>I find myself wanting to do a kind of “split” – that is, take a set of tuples, and do a join and an antijoin, and process the resulting tuples differently depending on which way they go. Is there an optimized way to do that?</p>
</blockquote>
<p>The answer should be yes. Let me try talking it out to convince myself. This is the implementation of <code>antijoin</code>:</p>
<div class="codehilite"><pre><span></span><span class="w">        </span><span class="bp">self</span><span class="p">.</span><span class="n">concat</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">.</span><span class="n">semijoin</span><span class="p">(</span><span class="n">other</span><span class="p">).</span><span class="n">negate</span><span class="p">())</span><span class="w"></span>
</pre></div>


<p>so it computes <code>self.semijoin(other)</code> as part of its definition. Essentially it determines an antijoin by taking the original stream and subtracting off the semijoined terms. You could totally do the same thing by hand:</p>
<div class="codehilite"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">semi</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rel1</span><span class="p">.</span><span class="n">semijoin</span><span class="p">(</span><span class="o">&amp;</span><span class="n">rel2</span><span class="p">);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">anti</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">semi</span><span class="p">.</span><span class="n">negate</span><span class="p">().</span><span class="n">concat</span><span class="p">(</span><span class="o">&amp;</span><span class="n">rel1</span><span class="p">);</span><span class="w"></span>
</pre></div>


<p>This does no more work than <code>antijoin</code> and gives you a handle to both <code>semi</code> and <code>anti</code>, which should be what you want. </p>
<p>Important caveat: <code>antijoin</code> only works correctly under the assumption that <code>rel2</code> has maximum cardinality one; the <code>semijoin</code> implementation is not a restriction so much as a multiplication, so if <code>rel2</code> has records with multiplicity 2 or more, the antijoin logic is messed up. This can always be fixed with a <code>distinct</code>, but it isn't done automatically in case you have other ways to guarantee that distinctness holds.</p>



<a name="126459979"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126459979" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126459979">(May 12 2018 at 12:10)</a>:</h4>
<p>(if it helps visualize the datalog, this is a graph of the Naive rules) <a href="/user_uploads/4715/eRWgvQafdfpsTxE6xWK2lIqK/Naive-precedence-graph.png" target="_blank" title="Naive-precedence-graph.png">Naive-precedence-graph.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/eRWgvQafdfpsTxE6xWK2lIqK/Naive-precedence-graph.png" target="_blank" title="Naive-precedence-graph.png"><img src="/user_uploads/4715/eRWgvQafdfpsTxE6xWK2lIqK/Naive-precedence-graph.png"></a></div>



<a name="126460142"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126460142" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126460142">(May 12 2018 at 12:17)</a>:</h4>
<p>Ooo, neat.</p>



<a name="126460235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126460235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126460235">(May 12 2018 at 12:20)</a>:</h4>
<p>(this is generated by soufflé's debug-report) it could be neat to have a graph of the timely computation, the different operators and so on</p>



<a name="126460240"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126460240" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126460240">(May 12 2018 at 12:20)</a>:</h4>
<p>Some more data from <code>TimelyOpt</code>: the new <code>subset</code> computation takes about 16s and produces 2,714,121 distinct facts. The <code>requires</code> computation then takes 35s longer and produces 9,585,213 distinct facts. It produces about 30m facts pre-distinctness, which isn't massively out of line with a 1us per fact rate that other computations are doing. If there really are 9.5M result <code>requires</code> tuples needed, it might be that it is time to start engineering the computation rather than the query.</p>



<a name="126460251"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126460251" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126460251">(May 12 2018 at 12:21)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> Yeah, we've had that in the past. The logging machinery logs all dataflow creation events, and you can feed them in to a visualizer of your choosing (Andrea Lattuada had one that would produce static graphs). You .. can end up with a bit of a tangle, though. :)</p>



<a name="126460299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126460299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126460299">(May 12 2018 at 12:22)</a>:</h4>
<p>That picture is much prettier (simpler, let's say) than what we usually produce (with all the <code>map</code> and <code>concat</code> and loop feedback nodes and edges).</p>



<a name="126460300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126460300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126460300">(May 12 2018 at 12:22)</a>:</h4>
<p>:)</p>



<a name="126462062"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462062" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462062">(May 12 2018 at 13:32)</a>:</h4>
<p>still trying out soufflé with the Naive rules, its profiler is cool, eg:</p>
<p>Relations — sorted by Total time, desc</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>ID</th>
<th>Total Time</th>
<th>Non Rec Time</th>
<th>Rec Time</th>
<th>Copy Time</th>
<th>Tuples</th>
<th>% of Time</th>
<th>% of Tuples</th>
</tr>
</thead>
<tbody>
<tr>
<td>subset</td>
<td>R1</td>
<td>16.07m</td>
<td>0.056s</td>
<td>16.03m</td>
<td>2.31s</td>
<td>7.53M</td>
<td>92.3</td>
<td>42.0</td>
</tr>
<tr>
<td>requires</td>
<td>R2</td>
<td>78.9s</td>
<td>0.002s</td>
<td>72.9s</td>
<td>6.06s</td>
<td>9.59M</td>
<td>7.56</td>
<td>53.4</td>
</tr>
<tr>
<td>borrow_live_at</td>
<td>R3</td>
<td>1.43s</td>
<td>1.43s</td>
<td>0</td>
<td>0</td>
<td>832K</td>
<td>0.137</td>
<td>4.64</td>
</tr>
</tbody>
</table>
<p>Rules — sorted by Total time, desc</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>ID</th>
<th>Total Time</th>
<th>Non Rec Time</th>
<th>Rec Time</th>
<th>Copy Time</th>
<th>Tuples</th>
<th>% of Time</th>
<th>% of Tuples</th>
</tr>
</thead>
<tbody>
<tr>
<td>subset(R1,R3,P) :-     subset(R1,R2,P),    subset(R2,R3,P).</td>
<td>C1.1</td>
<td>15.80m</td>
<td>0</td>
<td>15.75m</td>
<td>2.51s</td>
<td>4.97M</td>
<td>90.8</td>
<td>27.7</td>
</tr>
<tr>
<td>requires(R2,B,P) :-     requires(R1,B,P),    subset(R1,R2,P).</td>
<td>C2.1</td>
<td>56.4s</td>
<td>0</td>
<td>53.6s</td>
<td>2.74s</td>
<td>5.42M</td>
<td>5.40</td>
<td>30.2</td>
</tr>
<tr>
<td>requires(R,B,Q) :-     requires(R,B,P),    cfg_edge(P,Q),    region_live_at(R,Q),    !killed(B,P).</td>
<td>C2.2</td>
<td>12.2s</td>
<td>0</td>
<td>10.6s</td>
<td>1.54s</td>
<td>3.06M</td>
<td>1.16</td>
<td>17.0</td>
</tr>
<tr>
<td>requires(R,B,Q) :-     requires(R,B,P),    cfg_edge(P,Q),    universal_region(R),    !killed(B,P).</td>
<td>C2.3</td>
<td>9.08s</td>
<td>0</td>
<td>8.52s</td>
<td>0.556s</td>
<td>1.10M</td>
<td>0.869</td>
<td>6.14</td>
</tr>
<tr>
<td>subset(R1,R2,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    region_live_at(R1,Q),    region_live_at(R2,Q).</td>
<td>C1.2</td>
<td>6.37s</td>
<td>0</td>
<td>5.45s</td>
<td>0.921s</td>
<td>1.82M</td>
<td>0.610</td>
<td>10.2</td>
</tr>
<tr>
<td>subset(R1,R2,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    region_live_at(R1,Q),    universal_region(R2).</td>
<td>C1.3</td>
<td>3.89s</td>
<td>0</td>
<td>3.81s</td>
<td>0.076s</td>
<td>151K</td>
<td>0.372</td>
<td>0.843</td>
</tr>
<tr>
<td>subset(R1,R2,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    universal_region(R1),    region_live_at(R2,Q).</td>
<td>C1.4</td>
<td>3.67s</td>
<td>0</td>
<td>3.65s</td>
<td>0.021s</td>
<td>42.4K</td>
<td>0.352</td>
<td>0.236</td>
</tr>
<tr>
<td>subset(R1,R2,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    universal_region(R1),    universal_region(R2).</td>
<td>C1.5</td>
<td>3.32s</td>
<td>0</td>
<td>3.32s</td>
<td>0.006s</td>
<td>11.2K</td>
<td>0.318</td>
<td>0.062</td>
</tr>
<tr>
<td>borrow_live_at(P,B) :-     requires(R,B,P),    region_live_at(R,P).</td>
<td>N3.1</td>
<td>1.09s</td>
<td>1.09s</td>
<td>0</td>
<td>0</td>
<td>803K</td>
<td>0.104</td>
<td>4.47</td>
</tr>
<tr>
<td>borrow_live_at(P,B) :-     requires(R,B,P),    universal_region(R).</td>
<td>N3.2</td>
<td>0.325s</td>
<td>0.325s</td>
<td>0</td>
<td>0</td>
<td>29.8K</td>
<td>0.031</td>
<td>0.166</td>
</tr>
<tr>
<td>subset(R1,R2,P) :-     outlives(R1,R2,P).</td>
<td>N1.1</td>
<td>0.056s</td>
<td>0.056s</td>
<td>0</td>
<td>0</td>
<td>534K</td>
<td>0.005</td>
<td>2.98</td>
</tr>
<tr>
<td>requires(R,B,P) :-     borrow_region(R,B,P).</td>
<td>N2.1</td>
<td>0.002s</td>
<td>0.002s</td>
<td>0</td>
<td>0</td>
<td>1.89K</td>
<td>0.000</td>
<td>0.011</td>
</tr>
</tbody>
</table>



<a name="126462069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462069">(May 12 2018 at 13:33)</a>:</h4>
<p>(yes it takes 17mins over the clap facts on this laptop :p) I'll try and get some numbers from the TimelyOpt rules</p>



<a name="126462166"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462166" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462166">(May 12 2018 at 13:38)</a>:</h4>
<p>That is super cool (the souffle profiling). 17 mins is mental though. DD takes less time even if you count the build time. :D  Is that the compiled mode, or interpreted, <span class="user-mention" data-user-id="116113">@lqd</span>?</p>



<a name="126462211"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462211" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462211">(May 12 2018 at 13:39)</a>:</h4>
<p>it's in compiled mode, and with 2 threads ;) but as niko said, it's just the original rule ordering, ie reordering it better would surely make it a lot faster — but with a 15mins feedback loop I'm not even going to look into that :p</p>



<a name="126462254"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462254" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462254">(May 12 2018 at 13:40)</a>:</h4>
<p>niko mentioned having seen 10x better times from reordering</p>



<a name="126462255"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462255" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462255">(May 12 2018 at 13:40)</a>:</h4>
<p>Also, more measurements and thoughts on <code>TimelyOpt</code>. The <code>borrow_live_at</code> relation, which I <em>think</em> is the main output of the analysis, has only 832,392 output facts, which is a substantial reduction from the 10M or so from <code>requires</code>. The rule that produces it is</p>
<div class="codehilite"><pre><span></span>    // borrow_live_at(B, P) :- requires(R, B, P), region_live_at(R, P)
</pre></div>


<p>which makes me think that there is the potential to short-circuit a bunch of the <code>requires</code> derivations. It isn't obvious how to do this yet, but it seems reasonable that we might be able to re-write the <code>requires</code> derivation so that each <code>borrow_live_at</code> fact acts as an antibody and suppresses further derivations of the fact (though, we have to avoid interfering with other derivations).</p>



<a name="126462298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462298">(May 12 2018 at 13:42)</a>:</h4>
<p>(I mostly wanted to see time %ages, and tuple %ages and absolute counts, per rule)</p>



<a name="126462304"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462304" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462304">(May 12 2018 at 13:43)</a>:</h4>
<p>(and for comparison the Naive DD runs in 200-240s on the same machine)</p>



<a name="126462406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462406">(May 12 2018 at 13:47)</a>:</h4>
<p>Gah, Niko has disabled multi-threaded execution here:</p>
<div class="codehilite"><pre><span></span><span class="w">    </span><span class="n">timely</span>::<span class="n">execute_from_args</span><span class="p">(</span><span class="n">vec</span><span class="o">!</span><span class="p">[].</span><span class="n">into_iter</span><span class="p">(),</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="p">...</span><span class="w"></span>
</pre></div>


<p>That would probably bump up perf a bit more.</p>



<a name="126462587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462587">(May 12 2018 at 13:54)</a>:</h4>
<p>Souffle's profiling info also looks better than timely/differential's:</p>
<p><a href="/user_uploads/4715/mAkPxPVyVCvijhl9IuQ9e25U/TimelyOpt-profiling.png" target="_blank" title="TimelyOpt-profiling.png">TimelyOpt-profiling.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/mAkPxPVyVCvijhl9IuQ9e25U/TimelyOpt-profiling.png" target="_blank" title="TimelyOpt-profiling.png"><img src="/user_uploads/4715/mAkPxPVyVCvijhl9IuQ9e25U/TimelyOpt-profiling.png"></a></div>



<a name="126462637"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462637" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462637">(May 12 2018 at 13:56)</a>:</h4>
<p>About a 10% perf bump using github master DD, fwiw. (vs <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a> version).</p>



<a name="126462733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462733">(May 12 2018 at 14:00)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span>  here's an upload of its html profiler for you, if you wanted to check it out: <a href="https://lqd.github.io/profiler_html/main.html" target="_blank" title="https://lqd.github.io/profiler_html/main.html">https://lqd.github.io/profiler_html/main.html</a></p>



<a name="126462741"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462741" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462741">(May 12 2018 at 14:01)</a>:</h4>
<p><em>steals</em></p>



<a name="126462783"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462783" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462783">(May 12 2018 at 14:02)</a>:</h4>
<p>(the data is separate, it's a json file, very easy to steal :p)</p>



<a name="126462804"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126462804" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126462804">(May 12 2018 at 14:04)</a>:</h4>
<p>The ETHZ group has had a mind to do a D3 attachment that takes the timely logs (they stream out too) into a live dataflow graph, so you can watch the computation and see these sorts of stats live.  Just need to find someone who wants to spend the time. &gt;.&lt;</p>



<a name="126468056"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126468056" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126468056">(May 12 2018 at 17:26)</a>:</h4>
<p>Just in case it's useful to anyone, here's some info about the (tentative) datalog soufflé version of TimelyOpt, with precedence graph <a href="/user_uploads/4715/SgvxtEtYrY3TJ-WwHs4mcs5s/timelyopt-precedence-graph.png" target="_blank" title="timelyopt-precedence-graph.png">timelyopt-precedence-graph.png</a> and SCC graph <a href="/user_uploads/4715/exrvXPvoVTJp9N9FvM2fPRcW/timelyopt-scc-graph.png" target="_blank" title="timelyopt-scc-graph.png">timelyopt-scc-graph.png</a>.</p>
<div class="message_inline_image"><a href="/user_uploads/4715/SgvxtEtYrY3TJ-WwHs4mcs5s/timelyopt-precedence-graph.png" target="_blank" title="timelyopt-precedence-graph.png"><img src="/user_uploads/4715/SgvxtEtYrY3TJ-WwHs4mcs5s/timelyopt-precedence-graph.png"></a></div><div class="message_inline_image"><a href="/user_uploads/4715/exrvXPvoVTJp9N9FvM2fPRcW/timelyopt-scc-graph.png" target="_blank" title="timelyopt-scc-graph.png"><img src="/user_uploads/4715/exrvXPvoVTJp9N9FvM2fPRcW/timelyopt-scc-graph.png"></a></div><p>And a profile, for tuple numbers &amp; relative %ages:<br>
Relations — sorted by Total time, desc</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>ID</th>
<th>Total Time</th>
<th>Non Rec Time</th>
<th>Rec Time</th>
<th>Copy Time</th>
<th>Tuples</th>
<th>% of Time</th>
<th>% of Tuples</th>
</tr>
</thead>
<tbody>
<tr>
<td>requires</td>
<td>R4</td>
<td>52.8s</td>
<td>268.8µs</td>
<td>46.3s</td>
<td>6.50s</td>
<td>9.59M</td>
<td>50.2</td>
<td>71.9</td>
</tr>
<tr>
<td>dead_can_reach</td>
<td>R3</td>
<td>36.1s</td>
<td>0</td>
<td>36.1s</td>
<td>0.074s</td>
<td>170K</td>
<td>34.4</td>
<td>1.27</td>
</tr>
<tr>
<td>subset</td>
<td>R1</td>
<td>12.6s</td>
<td>0.061s</td>
<td>11.4s</td>
<td>1.13s</td>
<td>2.70M</td>
<td>12.0</td>
<td>20.3</td>
</tr>
<tr>
<td>live_to_dead_regions</td>
<td>R2</td>
<td>2.01s</td>
<td>0</td>
<td>2.01s</td>
<td>0.008s</td>
<td>38.1K</td>
<td>1.92</td>
<td>0.286</td>
</tr>
<tr>
<td>borrow_live_at</td>
<td>R5</td>
<td>1.53s</td>
<td>1.53s</td>
<td>0</td>
<td>0</td>
<td>832K</td>
<td>1.46</td>
<td>6.25</td>
</tr>
</tbody>
</table>
<p>Rules — sorted by Total time, desc</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>ID</th>
<th>Total Time</th>
<th>Non Rec Time</th>
<th>Rec Time</th>
<th>Copy Time</th>
<th>Tuples</th>
<th>% of Time</th>
<th>% of Tuples</th>
</tr>
</thead>
<tbody>
<tr>
<td>dead_can_reach(R1,R3,P,Q) :-     dead_can_reach(R1,R2,P,Q),    subset(R2,R3,P),    !region_live_at(R2,Q),    !universal_region(R2).</td>
<td>C3.2</td>
<td>30.8s</td>
<td>0</td>
<td>30.8s</td>
<td>0.061s</td>
<td>94.2K</td>
<td>29.5</td>
<td>0.706</td>
</tr>
<tr>
<td>requires(R2,B,P) :-     requires(R1,B,P),    subset(R1,R2,P).</td>
<td>C4.1</td>
<td>30.3s</td>
<td>0</td>
<td>24.7s</td>
<td>5.63s</td>
<td>8.73M</td>
<td>29.0</td>
<td>65.5</td>
</tr>
<tr>
<td>requires(R,B,Q) :-     requires(R,B,P),    cfg_edge(P,Q),    region_live_at(R,Q),    !killed(B,P).</td>
<td>C4.2</td>
<td>12.4s</td>
<td>0</td>
<td>11.9s</td>
<td>0.520s</td>
<td>807K</td>
<td>11.8</td>
<td>6.05</td>
</tr>
<tr>
<td>requires(R,B,Q) :-     requires(R,B,P),    cfg_edge(P,Q),    universal_region(R),    !killed(B,P).</td>
<td>C4.3</td>
<td>9.61s</td>
<td>0</td>
<td>9.57s</td>
<td>0.032s</td>
<td>50.1K</td>
<td>9.18</td>
<td>0.376</td>
</tr>
<tr>
<td>dead_can_reach(R2,R3,P,Q) :-     live_to_dead_regions(_R1,R2,P,Q),    subset(R2,R3,P).</td>
<td>C3.1</td>
<td>5.25s</td>
<td>0</td>
<td>5.20s</td>
<td>0.049s</td>
<td>75.8K</td>
<td>5.02</td>
<td>0.568</td>
</tr>
<tr>
<td>subset(R1,R2,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    region_live_at(R1,Q),    region_live_at(R2,Q).</td>
<td>C1.1</td>
<td>4.32s</td>
<td>0</td>
<td>2.93s</td>
<td>1.39s</td>
<td>2.15M</td>
<td>4.13</td>
<td>16.2</td>
</tr>
<tr>
<td>subset(R1,R3,Q) :-     live_to_dead_regions(R1,R2,P,Q),    dead_can_reach(R2,R3,P,Q),    region_live_at(R3,Q).</td>
<td>C1.5</td>
<td>2.97s</td>
<td>0</td>
<td>2.96s</td>
<td>0.010s</td>
<td>15.4K</td>
<td>2.84</td>
<td>0.116</td>
</tr>
<tr>
<td>subset(R1,R2,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    region_live_at(R1,Q),    universal_region(R2).</td>
<td>C1.2</td>
<td>2.04s</td>
<td>0</td>
<td>2.04s</td>
<td>0</td>
<td>0</td>
<td>1.95</td>
<td>0.000</td>
</tr>
<tr>
<td>subset(R1,R2,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    universal_region(R1),    region_live_at(R2,Q).</td>
<td>C1.3</td>
<td>1.70s</td>
<td>0</td>
<td>1.70s</td>
<td>11.6µs</td>
<td>18</td>
<td>1.62</td>
<td>0.000</td>
</tr>
<tr>
<td>subset(R1,R2,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    universal_region(R1),    universal_region(R2).</td>
<td>C1.4</td>
<td>1.63s</td>
<td>0</td>
<td>1.63s</td>
<td>0</td>
<td>0</td>
<td>1.56</td>
<td>0.000</td>
</tr>
<tr>
<td>borrow_live_at(P,B) :-     requires(R,B,P),    region_live_at(R,P).</td>
<td>N5.1</td>
<td>1.16s</td>
<td>1.16s</td>
<td>0</td>
<td>0</td>
<td>803K</td>
<td>1.11</td>
<td>6.02</td>
</tr>
<tr>
<td>live_to_dead_regions(R1,R2,P,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    region_live_at(R1,Q),    !region_live_at(R2,Q),    !universal_region(R2).</td>
<td>C2.1</td>
<td>1.14s</td>
<td>0</td>
<td>1.13s</td>
<td>0.011s</td>
<td>17.3K</td>
<td>1.09</td>
<td>0.130</td>
</tr>
<tr>
<td>live_to_dead_regions(R1,R2,P,Q) :-     subset(R1,R2,P),    cfg_edge(P,Q),    universal_region(R1),    !region_live_at(R2,Q),    !universal_region(R2).</td>
<td>C2.2</td>
<td>0.827s</td>
<td>0</td>
<td>0.813s</td>
<td>0.013s</td>
<td>20.8K</td>
<td>0.790</td>
<td>0.156</td>
</tr>
<tr>
<td>borrow_live_at(P,B) :-     requires(R,B,P),    universal_region(R).</td>
<td>N5.2</td>
<td>0.346s</td>
<td>0.346s</td>
<td>0</td>
<td>0</td>
<td>29.8K</td>
<td>0.331</td>
<td>0.224</td>
</tr>
<tr>
<td>subset(R1,R2,P) :-     outlives(R1,R2,P).</td>
<td>N1.1</td>
<td>0.061s</td>
<td>0.061s</td>
<td>0</td>
<td>0</td>
<td>534K</td>
<td>0.058</td>
<td>4.01</td>
</tr>
<tr>
<td>requires(R,B,P) :-     borrow_region(R,B,P).</td>
<td>N4.1</td>
<td>248.9µs</td>
<td>248.9µs</td>
<td>0</td>
<td>0</td>
<td>1.89K</td>
<td>0.000</td>
<td>0.014</td>
</tr>
</tbody>
</table>



<a name="126468169"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126468169" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126468169">(May 12 2018 at 17:31)</a>:</h4>
<p>(this is not taking into account niko's recent transitive loan propagation removal)</p>



<a name="126529389"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126529389" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126529389">(May 14 2018 at 09:14)</a>:</h4>
<blockquote>
<p>Some more data from <code>TimelyOpt</code>: the new <code>subset</code> computation takes about 16s and produces 2,714,121 distinct facts. The <code>requires</code> computation then takes 35s longer and produces 9,585,213 distinct facts. It produces about 30m facts pre-distinctness, which isn't massively out of line with a 1us per fact rate that other computations are doing. If there really are 9.5M result <code>requires</code> tuples needed, it might be that it is time to start engineering the computation rather than the query.</p>
</blockquote>
<p>I'm catching up here, but <span class="user-mention" data-user-id="116609">@Frank McSherry</span> how did you gather these numbers? Can you did it with my latest version as well (which is substantially faster)?</p>



<a name="126529405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126529405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126529405">(May 14 2018 at 09:15)</a>:</h4>
<blockquote>
<p>Niko has disabled multi-threaded execution here</p>
</blockquote>
<p>Oh, I did? =) I forgot about that. We should see what happens, though I think there's still plenty of room to improve on a single thread.</p>



<a name="126529464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126529464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126529464">(May 14 2018 at 09:17)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> </p>
<blockquote>
<p>Another optimization that the triangle example above points out, which could shave constant factors but not asymptotic terms, is to use the "arrangement" infrastructure to arrange each (collection, key) pair at most once. </p>
</blockquote>
<p>So I saw the "arrangement" infrastructure but I totally didn't understand it. Are there docs or examples you can point me at?</p>



<a name="126532488"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126532488" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126532488">(May 14 2018 at 10:58)</a>:</h4>
<p>I pulled the new code, and will try it out tonight. Iirc, things get a bit more complicated and I have a less good intuition about what is what, but I can attach the counters to the stream output and report back.</p>



<a name="126532548"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126532548" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126532548">(May 14 2018 at 11:00)</a>:</h4>
<p>Yeah, let's see. I have a pile of non-public code that demonstrates it.  I'll send you a copy via email (it .. relates to some other published work of people I don't want to antagonize). I'll also take a swing through the code and see if there are any opportunities for it; there were in the <code>Naive</code> code, and I may show you those changes to explain, even if that code is mostly dead.</p>



<a name="126534483"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126534483" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126534483">(May 14 2018 at 12:01)</a>:</h4>
<p>wouldn't it be interesting to have a way to see those counters, maybe displayed with the -v flag ?</p>



<a name="126534673"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126534673" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126534673">(May 14 2018 at 12:06)</a>:</h4>
<p>I can totally add that <span class="user-mention" data-user-id="116113">@lqd</span>. It's just an optional e.g.</p>
<div class="codehilite"><pre><span></span><span class="n">subset</span><span class="w"></span>
<span class="w">    </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="n">_</span><span class="o">|</span><span class="w"> </span><span class="p">())</span><span class="w"></span>
<span class="w">    </span><span class="p">.</span><span class="n">consolidate</span><span class="p">()</span><span class="w"></span>
<span class="w">    </span><span class="p">.</span><span class="n">inspect</span><span class="p">(</span><span class="o">|</span><span class="n">x</span><span class="o">|</span><span class="w"> </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">&quot;Subset counts: {:?}&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">x</span><span class="p">.</span><span class="mi">2</span><span class="p">));</span><span class="w"></span>
</pre></div>



<a name="126534698"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126534698" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126534698">(May 14 2018 at 12:07)</a>:</h4>
<p>(( Currently un-breaking something I broke in timely, so distracted, but can get around to that too ))</p>



<a name="126536457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126536457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126536457">(May 14 2018 at 13:01)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> thanks; I tried attaching the counters but had a hard time interpreting the output. But I was missing the <code>.map(|_| ()).consolidate()</code>, which I think would make all the difference (I was seeing way too many tuples)</p>



<a name="126536684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126536684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126536684">(May 14 2018 at 13:07)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> when I use the "arrange", would I want to arrange <em>both</em> things being joined?</p>



<a name="126536733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126536733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126536733">(May 14 2018 at 13:08)</a>:</h4>
<p>I guess that only makes sense if it is joined more than once, and often one side is an "ephemeral" result</p>



<a name="126536814"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126536814" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126536814">(May 14 2018 at 13:10)</a>:</h4>
<p>also, reading the code, I guess I could not do something like <code>.semijoin(&amp;arranged_region_live)</code> -- the code seems to call <code>arrange_by_self</code> on the result, and I don't see a "fast path" for <code>Arranged</code></p>



<a name="126536817"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126536817" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126536817">(May 14 2018 at 13:10)</a>:</h4>
<p>that is, <code>Arranged</code> doesn't implement <code>arrange_by_self</code> as a no-op</p>



<a name="126536820"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126536820" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126536820">(May 14 2018 at 13:10)</a>:</h4>
<p>so I guess I have to use <code>join_core</code> then?</p>



<a name="126536831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126536831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126536831">(May 14 2018 at 13:11)</a>:</h4>
<p>which admittedly doesn't seem too bad</p>



<a name="126538088"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126538088" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126538088">(May 14 2018 at 13:41)</a>:</h4>
<p>ok, I think I have it figured out</p>



<a name="126538489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126538489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126538489">(May 14 2018 at 13:50)</a>:</h4>
<p>that improved things to 14.2s; I updated the PR</p>



<a name="126540583"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126540583" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126540583">(May 14 2018 at 14:40)</a>:</h4>
<p>(fyi, just quickly hacking in a <code>-w 8</code> argument on my local machine seemed to be approx a 2x win -- 7.9s instead of 14s)</p>



<a name="126541165"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541165" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541165">(May 14 2018 at 14:53)</a>:</h4>
<blockquote>
<p>@Frank McSherry when I use the "arrange", would I want to arrange both things being joined?</p>
</blockquote>
<p>As you note, it's only helpful if there is re-use, but even if you don't arrange things the non-arranged <code>join</code> and such will internally do an arrange for you. They all call in to <code>join_core</code>, and put arranges in place if you haven't already done so yourself (and they do not re-arrange an arrangement, so shouldn't be worries there).</p>



<a name="126541202"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541202" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541202">(May 14 2018 at 14:53)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> yep, I kind of pieced it together from reading the source. Using <code>arrange</code> proved to be a small win. I kept it.</p>



<a name="126541209"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541209" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541209">(May 14 2018 at 14:53)</a>:</h4>
<p>That said, I'm not using it yet for <code>antijoin</code>, because I was too lazy :P</p>



<a name="126541271"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541271" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541271">(May 14 2018 at 14:54)</a>:</h4>
<p>as an aside, I've been thinking btw about how to write a "intro to differential dataflow" tutorial =) it'd be interesting to chat over with you at some point</p>



<a name="126541290"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541290" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541290">(May 14 2018 at 14:55)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> depending on your rig, it may be faster to use fewer workers than you have hardware threads. The timely workers are pretty impolite (and just sit on the cores) which can cause hiccups if the OS decides it has to do some things (one of the workers gets suspended, which can block parts of the compute).</p>



<a name="126541314"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541314" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541314">(May 14 2018 at 14:55)</a>:</h4>
<p>this is why we should add a flag so we can experiment with the number :) but that said I expect in the real compiler to have many simul. jobs, so it may not typically be worth using &gt;1 worker.</p>



<a name="126541322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541322">(May 14 2018 at 14:55)</a>:</h4>
<p>that said, I imagine we might want to make it depend on the size of the fn</p>



<a name="126541328"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541328" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541328">(May 14 2018 at 14:55)</a>:</h4>
<p>in any case I still think there's a lot of improvements to be made without jumping to parallelism</p>



<a name="126541409"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541409" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541409">(May 14 2018 at 14:57)</a>:</h4>
<p>Totally. Is it correct that the "output" for the computation is just the <code>borrow_live_at</code> relation? Is it important to compute all of <code>subset</code> and <code>requires</code> other than for determining <code>bla</code>, and might it be possible that derived <code>bla</code> tuples can suppress derivations in <code>subset</code> and <code>requires</code>?</p>



<a name="126541421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541421">(May 14 2018 at 14:57)</a>:</h4>
<p>heh, I was going to ask if timely used rayon work-stealing :)</p>



<a name="126541517"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541517" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541517">(May 14 2018 at 14:59)</a>:</h4>
<p><span class="user-mention" data-user-id="116013">@qmx</span> It doesn't (use rayon). It's a sane question to ask if that makes sense, but it conflicts a bit with the dataflow design elsewhere (e.g. the inter-operator shared state all needs mutexes, at least if done naively).</p>



<a name="126541615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541615">(May 14 2018 at 15:01)</a>:</h4>
<blockquote>
<p>Is it correct that the "output" for the computation is just the <code>borrow_live_at</code> relation?</p>
</blockquote>
<p>Mostly; in fact, I think we can make an even more narrow relation -- this is what <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> and <span class="user-mention" data-user-id="116106">@Reed Koser</span> have been working on in issue #4. Basically, we can narrow down to a set of points and loans where we need <code>borrow_live_at</code>.</p>
<p>That said, there is one other piece of the check that I have to think about, where we might wind up needing <code>subset</code> in some other cases. (But still not the full relation)</p>
<blockquote>
<p>Is it important to compute all of <code>subset</code> and <code>requires</code> other than for determining <code>bla</code></p>
</blockquote>
<p>No</p>
<blockquote>
<p>and might it be possible that derived <code>bla</code> tuples can suppress derivations in <code>subset</code> and <code>requires</code>?</p>
</blockquote>
<p>I think you are saying "is it possible that -- because <code>borrow_live_at</code> only cares about <em>some</em> region that requires a loan L -- we might save some time because right now we compute <em>all</em> regions that require that loan L?" If so, that might be true. I'm not really sure how much in practice now that I've suppressed the transitive computation. I suppose we could try to figure it out.</p>



<a name="126541683"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541683" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541683">(May 14 2018 at 15:02)</a>:</h4>
<p>I was also wondering whether we will save time just by knowing that we only care about a subset of loans in any case. But It's hard to say how often that will be true.</p>



<a name="126541710"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541710" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541710">(May 14 2018 at 15:03)</a>:</h4>
<p>In previous version (as of a day or two back) the size of <code>requires</code> was about 10x that of <code>borrow_live_at</code>, which made it seem like perhaps there were ways to thin down redundant derivations. Total speculation, though.</p>



<a name="126541795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541795">(May 14 2018 at 15:04)</a>:</h4>
<p>makes sense; it's worth checking again</p>



<a name="126541801"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126541801" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126541801">(May 14 2018 at 15:05)</a>:</h4>
<p>in any case I'd like to add some kind of <code>count</code> calls that can dump out the size of each relation</p>



<a name="126558215"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126558215" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126558215">(May 14 2018 at 21:04)</a>:</h4>
<p>"Regular Datalog" for incremental view maintenance from this paper <a href="https://arxiv.org/abs/1804.10565" target="_blank" title="https://arxiv.org/abs/1804.10565">https://arxiv.org/abs/1804.10565</a> could be interesting in our case (but maybe too limiting)</p>



<a name="126642898"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126642898" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126642898">(May 16 2018 at 12:34)</a>:</h4>
<p>I just wrote a Datalog-specific framework, not built on timely dataflow. It is a lot simpler, and could in principle be a lot faster. At the moment it is not actually faster, due to be being lazy about how I do merge sort. It's about 300 lines of code, and currently correctly does a reference reachability computation. The intent here is that if it ends up both i. faster and ii. easier to integrate into rustc, then great!</p>



<a name="126642955"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126642955" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126642955">(May 16 2018 at 12:36)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> I'm not sure what you are saying, is this a new project?</p>



<a name="126642958"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126642958" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126642958">(May 16 2018 at 12:36)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span>  I was looking at Jamie Brandon's older Imp ideas for the same reason if you remember his work (some with DD :)</p>



<a name="126642959"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126642959" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126642959">(May 16 2018 at 12:36)</a>:</h4>
<p>Yeah, brand new as of this morning.</p>



<a name="126642968"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126642968" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126642968">(May 16 2018 at 12:37)</a>:</h4>
<p>interesting. I was wondering whether a "pared down differential-dataflow" might do well for us</p>



<a name="126642974"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126642974" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126642974">(May 16 2018 at 12:37)</a>:</h4>
<p>Currently trying to sort out some performance issues (using <code>sort_unstable</code> increases running time by about 1.5x over <code>sort</code>).</p>



<a name="126643029"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643029" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643029">(May 16 2018 at 12:38)</a>:</h4>
<p>It's a super-simple implementation. I'll try and document it a bit more this evening. Currently does monotonic variables with join and map, but semijoin and antijoin should be easy.</p>



<a name="126643035"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643035" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643035">(May 16 2018 at 12:39)</a>:</h4>
<p>cool.</p>



<a name="126643095"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643095" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643095">(May 16 2018 at 12:40)</a>:</h4>
<p>Here's a gist at the moment: <a href="https://gist.github.com/frankmcsherry/60be7cfe54b5bf096e879c9d15d679e1" target="_blank" title="https://gist.github.com/frankmcsherry/60be7cfe54b5bf096e879c9d15d679e1">https://gist.github.com/frankmcsherry/60be7cfe54b5bf096e879c9d15d679e1</a></p>
<p>it's not well explained yet, but roughly: variables have a pile of facts, a "recent" pile of facts (new as of the last round) and a "to_add" pile of facts that should be introduced in the next round (and made recent). All of the operators look at variables as inputs, and produce new outputs "to_add".</p>



<a name="126643109"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643109" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643109">(May 16 2018 at 12:41)</a>:</h4>
<p>This part is what a program looks like:</p>
<div class="codehilite"><pre><span></span><span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">iteration</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Iteration</span>::<span class="n">new</span><span class="p">();</span><span class="w"></span>

<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">variable1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">iteration</span><span class="p">.</span><span class="n">variable</span>::<span class="o">&lt;</span><span class="p">(</span><span class="kt">u32</span><span class="p">,</span><span class="kt">u32</span><span class="p">)</span><span class="o">&gt;</span><span class="p">(</span><span class="s">&quot;nodes&quot;</span><span class="p">);</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">variable2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">iteration</span><span class="p">.</span><span class="n">variable</span>::<span class="o">&lt;</span><span class="p">(</span><span class="kt">u32</span><span class="p">,</span><span class="kt">u32</span><span class="p">)</span><span class="o">&gt;</span><span class="p">(</span><span class="s">&quot;edges&quot;</span><span class="p">);</span><span class="w"></span>

<span class="w">    </span><span class="n">variable1</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="n">nodes</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>
<span class="w">    </span><span class="n">variable2</span><span class="p">.</span><span class="n">insert</span><span class="p">(</span><span class="n">edges</span><span class="p">.</span><span class="n">into</span><span class="p">());</span><span class="w"></span>

<span class="w">    </span><span class="k">while</span><span class="w"> </span><span class="n">iteration</span><span class="p">.</span><span class="n">changed</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>

<span class="w">        </span><span class="c1">// N(a,c) &lt;-  N(a,b), E(b,c)</span>
<span class="w">        </span><span class="n">join_into</span><span class="p">(</span><span class="o">&amp;</span><span class="n">variable1</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="n">variable2</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="n">variable1</span><span class="p">,</span><span class="w"> </span><span class="o">|</span><span class="n">_b</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="n">c</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">c</span><span class="p">,</span><span class="n">a</span><span class="p">));</span><span class="w"></span>

<span class="w">    </span><span class="p">}</span><span class="w"></span>

<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">reachable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">variable1</span><span class="p">.</span><span class="n">complete</span><span class="p">();</span><span class="w"></span>
</pre></div>



<a name="126643179"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643179" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643179">(May 16 2018 at 12:43)</a>:</h4>
<p>the return of <code>gallop</code>:)</p>



<a name="126643186"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643186" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643186">(May 16 2018 at 12:43)</a>:</h4>
<p>It's the same one that Jamie used, fwiw. :)</p>



<a name="126643372"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643372" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643372">(May 16 2018 at 12:48)</a>:</h4>
<p>this might be not the end solution, but is immensely useful to me on the educational front to understand what's going on :)</p>



<a name="126643536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643536">(May 16 2018 at 12:53)</a>:</h4>
<p>I figure another advantage might be that, that this is a bit easier to grok, maintain, and debug (correctness/performance) if things go wrong (or if I get hit by a bus).</p>



<a name="126643537"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643537" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643537">(May 16 2018 at 12:53)</a>:</h4>
<p>Also, this reachability example at least takes 1.3s to compile in release mode. That might be welcome too. :)</p>



<a name="126643690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643690">(May 16 2018 at 12:56)</a>:</h4>
<p>I hadn't really considered this, but I love the idea of having a rust-lang-nursery crate that is just the core "datalog-y" framework that is owned by compiler team</p>



<a name="126643692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643692">(May 16 2018 at 12:56)</a>:</h4>
<p>it does address some of my other concerns about taking on differential-dataflow as a dependency</p>



<a name="126643698"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643698" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643698">(May 16 2018 at 12:56)</a>:</h4>
<p>e.g., I would prefer we keep rustc deps easily auditable</p>



<a name="126643712"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643712" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643712">(May 16 2018 at 12:57)</a>:</h4>
<p>it actually doesn't matter whether it's owned by rust-lang-nursery or not, but basically I just mean a project that can grow its own community</p>



<a name="126643715"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643715" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643715">(May 16 2018 at 12:57)</a>:</h4>
<p>and presumably be usable outside of just borrow-check</p>



<a name="126643720"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643720" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643720">(May 16 2018 at 12:57)</a>:</h4>
<p>I feel like a number of compiler optimizations etc might benefit from the same infrastructure</p>



<a name="126643783"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643783" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643783">(May 16 2018 at 12:58)</a>:</h4>
<p>obvious example; we compute liveness by hand with a home-grown thing</p>



<a name="126643796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126643796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126643796">(May 16 2018 at 12:59)</a>:</h4>
<p>(I have contemplated integrating that computaiton into borrow-check as well at some point...)</p>



<a name="126644465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126644465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126644465">(May 16 2018 at 13:15)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> the more I think about this the more I like it :) question: how long do you think it will take you to get something usable, <em>and</em>, would there be ways for us to help you :)</p>



<a name="126644507"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126644507" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126644507">(May 16 2018 at 13:16)</a>:</h4>
<p>"you're not alone man"</p>



<a name="126645282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645282">(May 16 2018 at 13:35)</a>:</h4>
<p>It's up and running. I'll push a version to github, and you can take a peek / comment on what's missing etc.</p>



<a name="126645285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645285">(May 16 2018 at 13:35)</a>:</h4>
<p>we definitely need <code>antijoin</code> etc</p>



<a name="126645287"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645287" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645287">(May 16 2018 at 13:35)</a>:</h4>
<p>though only I think on facts</p>



<a name="126645291"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645291" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645291">(May 16 2018 at 13:35)</a>:</h4>
<p>maybe I can see if I can ascertain how one might add it ;)</p>



<a name="126645293"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645293" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645293">(May 16 2018 at 13:35)</a>:</h4>
<p>do we name things <code>datalog-rs</code> or <code>datalog_rs</code> or what?</p>



<a name="126645298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645298">(May 16 2018 at 13:35)</a>:</h4>
<p>Oh, antijoin is super easy; I can do that, just lazy.</p>



<a name="126645299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645299">(May 16 2018 at 13:35)</a>:</h4>
<p>crate names should have <code>-</code></p>



<a name="126645342"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645342" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645342">(May 16 2018 at 13:36)</a>:</h4>
<blockquote>
<p>Oh, antijoin is super easy; I can do that, just lazy.</p>
</blockquote>
<p>I figured, it might just be interesting for me to contemplate how to do it :)</p>



<a name="126645343"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645343" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645343">(May 16 2018 at 13:36)</a>:</h4>
<p>in order to make myself understand code better</p>



<a name="126645345"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645345" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645345">(May 16 2018 at 13:36)</a>:</h4>
<p>although it looked prety clear from my skim</p>



<a name="126645352"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645352" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645352">(May 16 2018 at 13:36)</a>:</h4>
<p>(I don't know if I'll have time for that today anyway, i'm on a compressed schedule since I have to take my daughter to the hospital later)</p>



<a name="126645353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645353">(May 16 2018 at 13:36)</a>:</h4>
<p>Ah, in which case I can give you a chance. :)  There is some code with the signature commented out, but the logic in the body is totally not correct. (in the gist).</p>



<a name="126645358"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645358" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645358">(May 16 2018 at 13:36)</a>:</h4>
<blockquote>
<p>do we name things <code>datalog-rs</code> or <code>datalog_rs</code> or what?</p>
</blockquote>
<p>and it's recommended to not have "rs". I think Cargo will remove it automatically now</p>



<a name="126645359"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645359" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645359">(May 16 2018 at 13:36)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> heh don't feel the need to wait for me :)</p>



<a name="126645363"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645363" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645363">(May 16 2018 at 13:37)</a>:</h4>
<p>oh, that too</p>



<a name="126645380"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645380" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645380">(May 16 2018 at 13:37)</a>:</h4>
<p>but I think the github project can have it, just the cargo name typically doesn't</p>



<a name="126645382"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645382" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645382">(May 16 2018 at 13:37)</a>:</h4>
<p>for obvious reasons</p>



<a name="126645389"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645389" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645389">(May 16 2018 at 13:37)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> should I halt the other efforts in lieu of integrating this?</p>



<a name="126645396"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645396" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645396">(May 16 2018 at 13:38)</a>:</h4>
<p><span class="user-mention" data-user-id="116013">@qmx</span> I still think we want to do that factoring</p>



<a name="126645441"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645441" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645441">(May 16 2018 at 13:38)</a>:</h4>
<p>but probably not worth investigating how to get differential-dataflow into rustc</p>



<a name="126645443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645443">(May 16 2018 at 13:38)</a>:</h4>
<p>until we decide one way or the other here</p>



<a name="126645445"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645445" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645445">(May 16 2018 at 13:38)</a>:</h4>
<p>that is, we still want to factor borrow-check itself into a reusable library</p>



<a name="126645461"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645461" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645461">(May 16 2018 at 13:39)</a>:</h4>
<p>grr, I need to take a month-long vacation to play with rust</p>



<a name="126645470"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645470" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645470">(May 16 2018 at 13:39)</a>:</h4>
<p><span class="user-mention" data-user-id="116013">@qmx</span> do it do it</p>



<a name="126645472"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645472" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645472">(May 16 2018 at 13:39)</a>:</h4>
<p>/me looks at the clock</p>



<a name="126645535"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126645535" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126645535">(May 16 2018 at 13:40)</a>:</h4>
<p><a href="https://github.com/frankmcsherry/datalog-rs" target="_blank" title="https://github.com/frankmcsherry/datalog-rs">https://github.com/frankmcsherry/datalog-rs</a></p>



<a name="126646277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646277">(May 16 2018 at 13:58)</a>:</h4>
<p>What does <span class="emoji emoji-1f980" title="crab">:crab:</span> mean?</p>



<a name="126646281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646281">(May 16 2018 at 13:58)</a>:</h4>
<p>It's like the bat-sign, but for Rust code? :)</p>



<a name="126646299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646299">(May 16 2018 at 13:59)</a>:</h4>
<p><a href="http://www.rustacean.net/" target="_blank" title="http://www.rustacean.net/">http://www.rustacean.net/</a></p>



<a name="126646311"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646311" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646311">(May 16 2018 at 13:59)</a>:</h4>
<p>I use it to indicate ferris =)</p>



<a name="126646388"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646388" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646388">(May 16 2018 at 14:01)</a>:</h4>
<p>![ferris](<a href="http://www.rustacean.net/assets/rustacean-orig-noshadow.png" target="_blank" title="http://www.rustacean.net/assets/rustacean-orig-noshadow.png">http://www.rustacean.net/assets/rustacean-orig-noshadow.png</a>)</p>
<div class="message_inline_image"><a href="http://www.rustacean.net/assets/rustacean-orig-noshadow.png" target="_blank" title="http://www.rustacean.net/assets/rustacean-orig-noshadow.png"><img src="https://uploads.zulipusercontent.net/5e3c036eb2de043c5419db13f683e6c9c8ae3aac/687474703a2f2f7777772e72757374616365616e2e6e65742f6173736574732f72757374616365616e2d6f7269672d6e6f736861646f772e706e67"></a></div>



<a name="126646397"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646397" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646397">(May 16 2018 at 14:01)</a>:</h4>
<p>a Rusty high five ;)</p>



<a name="126646447"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646447" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646447">(May 16 2018 at 14:02)</a>:</h4>
<p>I was just trying to suss out the social implication. Like, if all of you had bowed your heads and clicked your hands as you scuttled side to side, you know "gotcha, but weird". :)</p>



<a name="126646463"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646463" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646463">(May 16 2018 at 14:02)</a>:</h4>
<p>Rusty high five is better. ;)   Antijoin is live, btw. Not tested yet, though.</p>



<a name="126646477"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646477" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646477">(May 16 2018 at 14:03)</a>:</h4>
<p>if we could configure this part of Zulip, I would make the <code>+</code> key add the <span class="emoji emoji-1f980" title="crab">:crab:</span> instead of <span class="emoji emoji-1f44d" title="+1">:+1:</span></p>



<a name="126646479"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646479" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646479">(May 16 2018 at 14:03)</a>:</h4>
<p>maybe I should open an issue</p>



<a name="126646531"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126646531" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126646531">(May 16 2018 at 14:04)</a>:</h4>
<p>Wow. + does do that!</p>



<a name="126648966"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126648966" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126648966">(May 16 2018 at 14:57)</a>:</h4>
<p>Repo updated with a hypothetical borrow checker (from naive, but without the universal region stuff). I haven't checked that it is correct yet, and so very bold to claim it would work. I did verify the reachability computation, but I'll try and get some facts loaded in and see if it produces the right number of tuples at least.</p>



<a name="126649015"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649015" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649015">(May 16 2018 at 14:58)</a>:</h4>
<p>(the zoidberg salute :3)</p>



<a name="126649028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649028">(May 16 2018 at 14:59)</a>:</h4>
<p>Also, you can see from the <code>borrow_check.rs</code> example,  <a href="https://github.com/frankmcsherry/datalog-rs/blob/master/src/bin/borrow_check.rs" target="_blank" title="https://github.com/frankmcsherry/datalog-rs/blob/master/src/bin/borrow_check.rs">https://github.com/frankmcsherry/datalog-rs/blob/master/src/bin/borrow_check.rs</a>, that the syntax can be a bit heavy. No method chaining at the moment, in the interest of keeping things lightweight (each call needs some more state managed, and we ask the programmer to do that manually rather than stashing it somewhere secret and managing it).</p>



<a name="126649127"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649127" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649127">(May 16 2018 at 15:01)</a>:</h4>
<p>if need be we can also generate calls to this API, parsed from datalog clauses</p>



<a name="126649128"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649128" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649128">(May 16 2018 at 15:01)</a>:</h4>
<p>I guess a key difference here is that you are not — as in differential-dataflow — modeling a program, but rather directly executing the actions</p>



<a name="126649131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649131">(May 16 2018 at 15:01)</a>:</h4>
<p>right?</p>



<a name="126649140"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649140" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649140">(May 16 2018 at 15:01)</a>:</h4>
<p>(hence the <code>while</code> and so forth)</p>



<a name="126649188"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649188" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649188">(May 16 2018 at 15:02)</a>:</h4>
<p>That is definitely a difference, yes. Each of the "direct executions" says "pick up any new records since last time around, and do what needs to be done wrt adding records to the output".</p>



<a name="126649195"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649195" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649195">(May 16 2018 at 15:02)</a>:</h4>
<p>but it does mean that someone needs to name the output, as the loop itself doesn't persist any state from iteration to iteration.</p>



<a name="126649210"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649210" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649210">(May 16 2018 at 15:03)</a>:</h4>
<p>I'm not sure what you mean by "name" yet</p>



<a name="126649220"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649220" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649220">(May 16 2018 at 15:03)</a>:</h4>
<p>Could probably clean this up a bit idiomatically, but could also let it be. It was pretty formulaic writing out the names of the intermediate relations, if tedious.</p>



<a name="126649221"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649221" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649221">(May 16 2018 at 15:03)</a>:</h4>
<div class="codehilite"><pre><span></span>subset_r1p.from_map(&amp;subset, |&amp;(r1,r2,p)| ((r1,p),r2));
</pre></div>



<a name="126649272"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649272" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649272">(May 16 2018 at 15:04)</a>:</h4>
<p>I guess <code>subset_r1p</code>  is the "named output"</p>



<a name="126649273"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649273" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649273">(May 16 2018 at 15:04)</a>:</h4>
<p>"name" meaning only "the operator itself does not create such a thing for you". So you need to point the operator at some place for it to put the results.</p>



<a name="126649274"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649274" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649274">(May 16 2018 at 15:04)</a>:</h4>
<p>you are referring to?</p>



<a name="126649280"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649280" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649280">(May 16 2018 at 15:04)</a>:</h4>
<p>Yes! The code is written as a lot of <code>relation.from_operator(...)</code> where the <code>relation</code> gets some more facts from the application of whatever the operator is.</p>



<a name="126649288"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649288" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649288">(May 16 2018 at 15:05)</a>:</h4>
<p>right, ok. I mean I think that's fine for now.</p>



<a name="126649294"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649294" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649294">(May 16 2018 at 15:05)</a>:</h4>
<p>it is still pretty easy to map to-from the underlying datalog</p>



<a name="126649299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649299">(May 16 2018 at 15:05)</a>:</h4>
<p>Old code was <code>rel1.join(&amp;rel2).join(&amp;rel3).map(..).blah(...)</code> where each step along the way the framework built the temporary collection to manage stuff; here the programmer has to do it.</p>



<a name="126649353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649353">(May 16 2018 at 15:07)</a>:</h4>
<p>yes I get it</p>



<a name="126649360"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649360" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649360">(May 16 2018 at 15:07)</a>:</h4>
<p>clearly the old code was nicer</p>



<a name="126649361"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649361" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649361">(May 16 2018 at 15:07)</a>:</h4>
<p>but this seems ok too</p>



<a name="126649366"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649366" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649366">(May 16 2018 at 15:07)</a>:</h4>
<p>we might be able to make some scheme where you create temporaries</p>



<a name="126649367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649367">(May 16 2018 at 15:07)</a>:</h4>
<p>that are not named</p>



<a name="126649368"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649368" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649368">(May 16 2018 at 15:07)</a>:</h4>
<p>well, I guess they have to persist across loop iterations</p>



<a name="126649369"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649369" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649369">(May 16 2018 at 15:07)</a>:</h4>
<p>so that doesn't work</p>



<a name="126649434"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649434" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649434">(May 16 2018 at 15:09)</a>:</h4>
<p>Yeah, some things like <code>map</code> could be simplified, but (at least in this implementation) you need a <code>Variable</code> wherever you want a collection indexed by some key so that you can do a join/antijoin.</p>



<a name="126649488"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649488" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649488">(May 16 2018 at 15:10)</a>:</h4>
<p>So for example the <code>subset</code> variable could probably be removed, as it contributes nothing over the three indexed variants. One can be produced, and the other two derived from it with <code>from_map()</code>.</p>



<a name="126649497"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649497" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649497">(May 16 2018 at 15:10)</a>:</h4>
<p>I was wondering whether this model allowed for us to optimize cases where we know something will be joined many times</p>



<a name="126649498"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649498" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649498">(May 16 2018 at 15:10)</a>:</h4>
<p>e.g. something like <code>arrange</code></p>



<a name="126649501"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649501" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649501">(May 16 2018 at 15:11)</a>:</h4>
<p>(I didn't read closely enough yet to see)</p>



<a name="126649505"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649505" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649505">(May 16 2018 at 15:11)</a>:</h4>
<p>Already does! :D</p>



<a name="126649515"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649515" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649515">(May 16 2018 at 15:11)</a>:</h4>
<p>For example, in the definition of <code>subset</code>, the two uses of <code>&amp;region_live_at</code> are just the same variable.</p>



<a name="126649564"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649564" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649564">(May 16 2018 at 15:12)</a>:</h4>
<p>/me looks</p>



<a name="126649573"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649573" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649573">(May 16 2018 at 15:12)</a>:</h4>
<p>I see</p>



<a name="126649586"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649586" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649586">(May 16 2018 at 15:13)</a>:</h4>
<p>This .. loses a lot of the flexibility of differential dataflow. The <code>Variable</code> types are roughly "two-version" relations, reflecting an "old" version and a "new" version. But because we know exactly when we want to use old and new, and when the entire world ticks forward from old to new (at the bottom of each loop), it is a bit easier to share the state.</p>



<a name="126649587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649587">(May 16 2018 at 15:13)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> are variables always (K, V) tuples?</p>



<a name="126649592"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649592" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649592">(May 16 2018 at 15:13)</a>:</h4>
<p>Variables are arbitrary <code>Tuple: Ord</code> types, and join applies when they are <code>(Key, Val)</code> pairs.</p>



<a name="126649679"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649679" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649679">(May 16 2018 at 15:16)</a>:</h4>
<p>I guess they don't even have to be tuples, so perhaps that is a misnomer.</p>



<a name="126649759"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649759" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649759">(May 16 2018 at 15:17)</a>:</h4>
<p>btw, (looking at the code) there are some weird perf gotchas. When we first create relations (sorted lists of tuples) <code>sort_unstable()</code> is faster than <code>sort()</code>, but when we merge relations then <code>sort()</code> is about 1.5x faster than <code>sort_unstable()</code>. Not sure what is up, but there are probably similar gotchas to watch out for perf-wise.</p>



<a name="126649766"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126649766" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126649766">(May 16 2018 at 15:17)</a>:</h4>
<p>On the plus side, can just point a profiler at it and see wtf is going on, and change it, I guess. :)</p>



<a name="126651868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126651868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126651868">(May 16 2018 at 16:02)</a>:</h4>
<p>The <code>datalog</code> crate name is taken, so working name is now <code>datafrog</code>. Complaints can be directed to .. well the maintainer of the <code>datalog</code> crate first, but then to /dev/null.</p>



<a name="126651910"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126651910" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126651910">(May 16 2018 at 16:02)</a>:</h4>
<p>I think I prefer <code>datafrog</code></p>



<a name="126651914"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126651914" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126651914">(May 16 2018 at 16:02)</a>:</h4>
<p>I see the logo already :P</p>



<a name="126651986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126651986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126651986">(May 16 2018 at 16:04)</a>:</h4>
<p><span class="emoji emoji-1f438" title="frog">:frog:</span></p>



<a name="126652015"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126652015" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126652015">(May 16 2018 at 16:05)</a>:</h4>
<p>even better, it's pre-drawn for you</p>



<a name="126652061"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126652061" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126652061">(May 16 2018 at 16:06)</a>:</h4>
<p>someone will expect leapfrog triejoin then :p</p>



<a name="126652111"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126652111" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126652111">(May 16 2018 at 16:08)</a>:</h4>
<p>Oh wow, good point <span class="user-mention" data-user-id="116113">@lqd</span></p>



<a name="126656227"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126656227" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126656227">(May 16 2018 at 17:45)</a>:</h4>
<p>I've pushed some doctests, it seems that in simple examples at least <code>join</code>, <code>antijoin</code>, and <code>map</code> work as planned. I'm actually flying to Vienna for some talks tomorrow morning (8pm here now) so not sure how much I'll get done wrt trying out <code>borrow_check.rs</code>. Happy to help out and put some time in when I get it, but I'll probably also take some time to check out Vienna too! :)</p>



<a name="126657281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126657281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126657281">(May 16 2018 at 18:09)</a>:</h4>
<p>This is all in that same repo ?</p>



<a name="126657350"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126657350" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126657350">(May 16 2018 at 18:10)</a>:</h4>
<p>yes</p>



<a name="126657355"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126657355" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126657355">(May 16 2018 at 18:10)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> bon voyage :)</p>



<a name="126674721"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126674721" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126674721">(May 17 2018 at 01:12)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> I've continued your borrowck-data <span class="emoji emoji-1f438" title="frog">:frog:</span> example: 1) loaded nll facts, 2) added <code>borrow_live_at</code>(but couldn't test it) 3) and begun testing the computation: the first iteration, computing <code>subset</code>, works and produces the same tuples as DD. However, the next iteration, for <code>requires</code> doesn't: I've surely made a mistake in the inputs or something, but I don't know where (that's also the reason why I couldn't test if <code>borrow_live_at</code> is correctly done or not). My branch is here <a href="https://github.com/lqd/borrow-check/tree/datafrog" target="_blank" title="https://github.com/lqd/borrow-check/tree/datafrog">https://github.com/lqd/borrow-check/tree/datafrog</a> — I've also temporarily disabled timely there so that it builds in 15s. I was testing with the smaller issue-47680 input, <code>requires</code>produces around 100 tuples instead of approx 150.</p>



<a name="126677092"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126677092" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126677092">(May 17 2018 at 02:40)</a>:</h4>
<p>I'm very pleased with "datafrog". Well done.</p>



<a name="126681867"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126681867" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126681867">(May 17 2018 at 05:37)</a>:</h4>
<p>I'll check it out <span class="user-mention" data-user-id="116113">@lqd</span>. One possible outcome (other than "you erred" or "I erred") is that it is computing the same answer, just differently. This implementation intentionally holds data back a bit longer to make sure everyone gets the same single view of the data, and that might mean that it takes more rounds for the facts to propagate (roughly the same amount of work though).</p>
<p>Ima investigate, but if you notice that it does indeed reach a different limit (different number of <code>bla</code> facts, for example) that is 100% evidence that something is wrong, whereas "second round different" might just be a warning sign.</p>



<a name="126682135"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126682135" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126682135">(May 17 2018 at 05:47)</a>:</h4>
<p>Ok, totally not the right total number of tuples:</p>
<div class="codehilite"><pre><span></span>Echidnatron% cargo +nightly run --release -- -a Datafrog inputs/clap-rs/app-parser-{{impl}}-add_defaults/
    Finished release [optimized] target(s) in 0.08s
     Running `target/release/borrow-check -a Datafrog &#39;inputs/clap-rs/app-parser-{{impl}}-add_defaults/&#39;`
subset is complete: 7531526
requires is complete: 84101
borrow_live_at is complete: 31933
--------------------------------------------------
Directory: inputs/clap-rs/app-parser-{{impl}}-add_defaults/
Time: 33.721s
</pre></div>



<a name="126682182"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126682182" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126682182">(May 17 2018 at 05:49)</a>:</h4>
<p>The count for subset looks "correct" (previously reported as 7.53M tuples), so will dive in to <code>requires</code> and what might be up there!</p>



<a name="126682235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126682235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126682235">(May 17 2018 at 05:50)</a>:</h4>
<p>Ah, one important thing (sorry!): the <code>borrow_check.rs</code> code that I hacked up (which looks like what you've based things on) totally ignored universal regions. I saw the issue that they were going away, and didn't port them for reasons of simplicity. But as long as the <code>region_live_at</code> relations don't yet have the universal regions in them that could be a source of discrepancy. Could be other issues too, of course. :D</p>



<a name="126682519"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126682519" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126682519">(May 17 2018 at 06:00)</a>:</h4>
<p>Further study (no answers yet) but it will almost 100% be the case that the number of tuples in each round of derivation are different from before. In particular each operator application takes one round to take effect, including easy operators like <code>from_map</code>. That shouldn't cause the computation to reach a different limit, so the glitchy outputs at the moment are still unresolved. Possible answers: i. bugs (datafrog loves bugs), ii. universal regions.</p>



<a name="126683110"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126683110" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126683110">(May 17 2018 at 06:20)</a>:</h4>
<p>Wait (reading your code more) you seem to have noticed the universal region thing and addressed it! I'm looking at the same issue you are looking at, and it seems small enough to get data out that are not what we expect. I'm not sure of the easiest way to get a dump of the legit <code>requires</code> facts from <code>-a Naive</code>, but if we have those and can start to debug the "when was this fact supposed to show up" process, that would be my next step.</p>



<a name="126683121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126683121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126683121">(May 17 2018 at 06:21)</a>:</h4>
<p>It does totally look like Datafrog bug though. Other options seem to have expired.</p>



<a name="126683283"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126683283" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126683283">(May 17 2018 at 06:26)</a>:</h4>
<p>Boarding in six minutes, so unlikely to solve right now. I can pick up the debugging later tonight, but next step is to find a tuple (or all 50) that should be derived but are not, so that we can check for the moment that should have happened and see what silly code prevented it.</p>



<a name="126685018"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126685018" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126685018">(May 17 2018 at 07:23)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> thank you for looking into this! I'll keep digging don't worry about it, enjoy the talks and Vienna :)  I was going to try and compare the <code>requires</code> subrules to what Native outputs (rn it's dumped with <code>-v</code>, as a list named <code>restricts</code> in the output). I'll try later today to get more information about the missing tuples. thanks again</p>



<a name="126685095"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126685095" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126685095">(May 17 2018 at 07:25)</a>:</h4>
<p>"my kingdom for why-not provenance"</p>



<a name="126686333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126686333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126686333">(May 17 2018 at 08:07)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> nice!</p>



<a name="126698751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126698751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126698751">(May 17 2018 at 14:03)</a>:</h4>
<p>My plan for this evening: grab the facts output from Datafrog and check that i. if you run the computation again you get no more facts (if it has reached fixed point you shouldn't, but maybe you do because bugs) and ii. when you input them into Souffle with the same rules you should get a few more facts (up to the ~150). These new facts are the ones that should be derived but are not, and should be easy to check out (with the same interned numbers as Datafrog).</p>
<p><span class="user-mention" data-user-id="116113">@lqd</span>, do you have a Souffle implementation of the Naive rules on hand? I know you had one version, once, but is it something you can check in to your repo?</p>



<a name="126698802"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126698802" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126698802">(May 17 2018 at 14:04)</a>:</h4>
<p>I've not had a chance to check out datafrog yet</p>



<a name="126698804"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126698804" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126698804">(May 17 2018 at 14:04)</a>:</h4>
<p>/me dying to do that</p>



<a name="126698805"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126698805" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126698805">(May 17 2018 at 14:04)</a>:</h4>
<p>sure, I have both the naive and opt one</p>



<a name="126698813"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126698813" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126698813">(May 17 2018 at 14:04)</a>:</h4>
<p>(I made a bit of progress I'll explain a bit later)</p>



<a name="126698830"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126698830" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126698830">(May 17 2018 at 14:05)</a>:</h4>
<p>Cool; between meetings now, and can check back in a few hours!</p>



<a name="126698991"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126698991" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126698991">(May 17 2018 at 14:08)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> can't commit rn, but here's a gist of both in the meantime <a href="https://gist.github.com/lqd/cb4c1e615be1eb0bcbf17919c0cef937" target="_blank" title="https://gist.github.com/lqd/cb4c1e615be1eb0bcbf17919c0cef937">https://gist.github.com/lqd/cb4c1e615be1eb0bcbf17919c0cef937</a> (the opt one worked in my tests but I'm not sure if I correctly translated the universal_region removal; no biggie since you wanted the naive one)</p>



<a name="126699001"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699001" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699001">(May 17 2018 at 14:09)</a>:</h4>
<p>Tyvm!</p>



<a name="126699011"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699011" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699011">(May 17 2018 at 14:09)</a>:</h4>
<p><span class="emoji emoji-1f438" title="frog">:frog:</span></p>



<a name="126699026"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699026" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699026">(May 17 2018 at 14:09)</a>:</h4>
<p>(datafrog is indeed a name as cool as Frank's previous "differential dataflog" :D)</p>



<a name="126699276"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699276" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699276">(May 17 2018 at 14:15)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> </p>
<div class="codehilite"><pre><span></span>Echidnatron% souffle -c ~/Projects/borrow-check/naive.dl | wc
     106     208    2241
Echidnatron%
</pre></div>



<a name="126699326"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699326" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699326">(May 17 2018 at 14:16)</a>:</h4>
<p>Are we sure that 150-ish is the right answer?</p>



<a name="126699338"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699338" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699338">(May 17 2018 at 14:17)</a>:</h4>
<p>I remember something like 135 now lemme check</p>



<a name="126699342"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699342" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699342">(May 17 2018 at 14:17)</a>:</h4>
<p>Oh wait, this is <code>bla</code> as output, and you are worried about <code>requires</code>. Nevermind me!</p>



<a name="126699505"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699505" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699505">(May 17 2018 at 14:20)</a>:</h4>
<p>think soufflé finds 152 requires</p>



<a name="126699570"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699570" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699570">(May 17 2018 at 14:22)</a>:</h4>
<p>I think the last of the requires rules is interesting, commenting out the rest basically gives the same output as DD</p>



<a name="126699593"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699593" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699593">(May 17 2018 at 14:23)</a>:</h4>
<p>the tuple I was about to try and locate was :<br>
timely (hopefully soufflé as well :p) has</p>
<div class="codehilite"><pre><span></span>&quot;Mid(bb2[0])&quot;    &quot;\&#39;_#6r&quot;         &quot;bw0&quot;
&quot;Mid(bb2[0])&quot;    &quot;\&#39;_#6r&quot;         &quot;bw2&quot;
</pre></div>



<a name="126699604"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699604" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699604">(May 17 2018 at 14:23)</a>:</h4>
<p>datafrog only has the bw0 one</p>



<a name="126699667"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699667" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699667">(May 17 2018 at 14:24)</a>:</h4>
<p>Cool; this is helpful! My plan is to try and get souffle going on the interned index values; do you think going backwards from the interned indices to strings is better (I'm not clear on how to do that)</p>



<a name="126699676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699676">(May 17 2018 at 14:25)</a>:</h4>
<p>I was just doing that rn</p>



<a name="126699702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699702">(May 17 2018 at 14:25)</a>:</h4>
<p>InternerTables "untern" could I think be used to get back from those strings to the indices</p>



<a name="126699760"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699760" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699760">(May 17 2018 at 14:26)</a>:</h4>
<p>but maybe using the interned values with soufflé would be less of a hassle I'm not sure</p>



<a name="126699769"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699769" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699769">(May 17 2018 at 14:26)</a>:</h4>
<p>Unless you strongly object, I'm just going to hack Datafrog to dump the interned tables as text, so that I can Souffle them up and use those values.</p>



<a name="126699786"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699786" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699786">(May 17 2018 at 14:27)</a>:</h4>
<p>oh I'm not objecting anything :D</p>



<a name="126699799"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699799" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699799">(May 17 2018 at 14:27)</a>:</h4>
<p>(don't take this sentence out of context)</p>



<a name="126699882"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126699882" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126699882">(May 17 2018 at 14:29)</a>:</h4>
<p>okies, off to meetings! I should get some dataz for you soon (well, later this evening).</p>



<a name="126700050"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126700050" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126700050">(May 17 2018 at 14:33)</a>:</h4>
<p>awesome :) (just a heads up I might not have access to a computer this evening)</p>



<a name="126706127"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126706127" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126706127">(May 17 2018 at 16:38)</a>:</h4>
<p>interestingly enough, all the 47 missing tuples from <code>requires</code> seem to be for the <code>bw2</code> loan</p>



<a name="126708901"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126708901" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126708901">(May 17 2018 at 17:35)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> I might be onto something, around the killed antijoin; killed is a &lt;L, P&gt; and it might need to be like the other semijoins, like &lt;(L, P), ()&gt;</p>



<a name="126709115"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126709115" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126709115">(May 17 2018 at 17:39)</a>:</h4>
<p>or maybe not :/ but something is up with this antijoin</p>



<a name="126710587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710587">(May 17 2018 at 18:05)</a>:</h4>
<p>some more results: hacking the worst antijoin of all time, joining <code>requires_bp((b,p),r)</code> into <code>requires_1</code> only if <code>(b, p)</code> is not in <code>killed</code>, we have:</p>
<div class="codehilite"><pre><span></span>subset is complete: 30
requires is complete: 152
borrow_live_at is complete: 102
</pre></div>


<p>of course these 102 data<span class="emoji emoji-1f438" title="frog">:frog:</span> <code>bla</code> are the same ones timely returns :3</p>



<a name="126710680"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710680" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710680">(May 17 2018 at 18:07)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> I don't understand that last line -- oh, bla is <code>borrow_live_at</code></p>



<a name="126710686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710686">(May 17 2018 at 18:07)</a>:</h4>
<p>yes sorry</p>



<a name="126710687"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710687" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710687">(May 17 2018 at 18:07)</a>:</h4>
<p>so basically <code>antijoin</code> is the problem, definitely</p>



<a name="126710696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710696">(May 17 2018 at 18:07)</a>:</h4>
<p>probably more my inputs to it I think</p>



<a name="126710755"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710755" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710755">(May 17 2018 at 18:08)</a>:</h4>
<p>but there's a mismatch between what frank's requires_1 expected (eg a relation) vs what i could be (if it's the semijoin structure I mentioned)</p>



<a name="126710778"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710778" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710778">(May 17 2018 at 18:09)</a>:</h4>
<p>it's definitely around this predicate for sure tho, so that's good to know :)</p>



<a name="126710786"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710786" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710786">(May 17 2018 at 18:09)</a>:</h4>
<p>we gotta rename <code>borrow_live_at</code> to <code>loan_live_at</code>, so that the acronym is <code>lla</code> :)</p>



<a name="126710792"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710792" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710792">(May 17 2018 at 18:09)</a>:</h4>
<p>/me gets confused each time he sees "bla"</p>



<a name="126710795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126710795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126710795">(May 17 2018 at 18:09)</a>:</h4>
<p>and all Bs to Ls !</p>



<a name="126712198"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712198" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712198">(May 17 2018 at 18:40)</a>:</h4>
<p>tried it with clap just to check the number of tuples and we're at 832392 <code>borrow_live_at</code> like we expect <span class="emoji emoji-1f389" title="tada">:tada:</span>  (I don't want to talk about how long it took  :p)</p>



<a name="126712636"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712636" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712636">(May 17 2018 at 18:51)</a>:</h4>
<p>that's with the "worst antijoin ever"?</p>



<a name="126712700"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712700" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712700">(May 17 2018 at 18:52)</a>:</h4>
<p>yes indeed, and more copies/indexing than necessary (as I was trying to stay as close to the initial rules) as some of the indexing work can be shared between the 1st 2 "iterations" (as was the case with frank's original example btw)</p>



<a name="126712727"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712727" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712727">(May 17 2018 at 18:53)</a>:</h4>
<p>for the smaller input it's very competitive even in this state</p>



<a name="126712806"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712806" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712806">(May 17 2018 at 18:55)</a>:</h4>
<p>/me facepalms</p>



<a name="126712813"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712813" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712813">(May 17 2018 at 18:55)</a>:</h4>
<p>lemme check something real quick lol</p>



<a name="126712887"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712887" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712887">(May 17 2018 at 18:57)</a>:</h4>
<p>ok I take it back I do want to talk about it</p>



<a name="126712946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712946">(May 17 2018 at 18:58)</a>:</h4>
<p>with all the caveats I mentioned before, we're at 60s naive data-<span class="emoji emoji-1f438" title="frog">:frog:</span> vs 148s for naive DD</p>



<a name="126712992"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712992" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712992">(May 17 2018 at 18:59)</a>:</h4>
<p>of course among the long command line, sorting &amp; uniquing and diffing the output, of course the "--release" characters were missing</p>



<a name="126712999"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126712999" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126712999">(May 17 2018 at 18:59)</a>:</h4>
<p>that's awesome!</p>



<a name="126713106"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126713106" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126713106">(May 17 2018 at 19:01)</a>:</h4>
<p>sounds good for a future <span class="emoji emoji-1f438" title="frog">:frog:</span>-opt (which I'll do tomorrow if nobody beats me to it, as I'll be mobile-only tonight) -- frank will know how to deal with the antijoin</p>



<a name="126713179"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126713179" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126713179">(May 17 2018 at 19:02)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> did I also mention polonius with datafrog compiles in 8s vs 2.5mins with timely ? :)</p>



<a name="126713195"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126713195" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126713195">(May 17 2018 at 19:03)</a>:</h4>
<p>(on this beefy machine, at home it's like 10 mins)</p>



<a name="126713300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126713300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126713300">(May 17 2018 at 19:05)</a>:</h4>
<p>(the antijoin is sooner in the datafrog example compared to naive DD, and it doesn't work the same way so of course it's a "grain of salt" situation for this comparison)</p>



<a name="126713657"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126713657" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126713657">(May 17 2018 at 19:12)</a>:</h4>
<p>(there might be some possible parallelism to have when creating tuples in the joins each round, à la "R×S = (R delta × S) + (R × S delta) + (R delta × S delta)")</p>



<a name="126714657"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126714657" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126714657">(May 17 2018 at 19:35)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> can't wait, compile times have been killin' me</p>



<a name="126715761"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126715761" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126715761">(May 17 2018 at 20:01)</a>:</h4>
<p>This all looks really good. I can believe the antijoin is glitchy, but not obvious to me at the moment what is wrong. The argument is a <code>Relation</code> to try and ensure that no one uses a variable (that might grow with iteration); .. that shouldn't make it wrong, but .. it could lead to bugs? I'm not sure.</p>



<a name="126715814"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126715814" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126715814">(May 17 2018 at 20:02)</a>:</h4>
<p>I'll try and get at this tomorrow. Like a moron, I left my power cable at IST in Vienna, so it won't be until tomorrow that I get back there, and on battery power for now.</p>



<a name="126715897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126715897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126715897">(May 17 2018 at 20:04)</a>:</h4>
<p>sad that there is no "crying frog" emoji</p>



<a name="126716172"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716172" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716172">(May 17 2018 at 20:10)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> I'm on mobile but here's teh ugliness <a href="https://gist.github.com/lqd/c2c645ac053ecc8340ca12e8b0f5d03b" target="_blank" title="https://gist.github.com/lqd/c2c645ac053ecc8340ca12e8b0f5d03b">https://gist.github.com/lqd/c2c645ac053ecc8340ca12e8b0f5d03b</a></p>



<a name="126716178"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716178" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716178">(May 17 2018 at 20:10)</a>:</h4>
<p>Ah sweet; was just about to beg for that. :D</p>



<a name="126716212"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716212" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716212">(May 17 2018 at 20:11)</a>:</h4>
<p>TIL copying big files on a phone, not that easy</p>



<a name="126716282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716282">(May 17 2018 at 20:13)</a>:</h4>
<p>Ok, I'll look into this. I've got some code that is <em>super similar</em> but still producing the wrong answer. Not at all clear why.</p>



<a name="126716287"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716287" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716287">(May 17 2018 at 20:13)</a>:</h4>
<p>I have full confidence in your laptop, it has bested many big data clusters before <span class="emoji emoji-1f609" title="wink">:wink:</span></p>



<a name="126716346"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716346" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716346">(May 17 2018 at 20:14)</a>:</h4>
<p>Well, it goes plenty fast on all of these computations, just gets the wrong answer. Didn't say anything about it being more accurate than the clusters. ;)</p>



<a name="126716361"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716361" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716361">(May 17 2018 at 20:14)</a>:</h4>
<p><span class="emoji emoji-1f642" title="simple smile">:simple_smile:</span></p>



<a name="126716379"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716379" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716379">(May 17 2018 at 20:15)</a>:</h4>
<p>Also, yours is not the worst antijoin of all time; mine is basically the same but uses <code>Vec::contains()</code>.</p>



<a name="126716426"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716426" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716426">(May 17 2018 at 20:16)</a>:</h4>
<p>I had that before as well !</p>



<a name="126716442"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716442" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716442">(May 17 2018 at 20:17)</a>:</h4>
<p>But mine gives also gives the wrong answer, so I have you beat there!</p>



<a name="126716491"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716491" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716491">(May 17 2018 at 20:18)</a>:</h4>
<p>everything's a competition I see how it is <span class="emoji emoji-1f603" title="smiley">:smiley:</span></p>



<a name="126716514"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716514" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716514">(May 17 2018 at 20:19)</a>:</h4>
<p>Want to know something crazy: my antijoin <em>isn't suppressing any tuples</em>. Everything it sees it emits as output. Somehow it isn't seeing the right inputs, or something...</p>



<a name="126716596"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716596" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716596">(May 17 2018 at 20:21)</a>:</h4>
<p>it must be doing something right, the test looked sane IIRC</p>



<a name="126716670"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716670" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716670">(May 17 2018 at 20:23)</a>:</h4>
<p>that's also why I was wondering whether it was the input, which didn't look like the other semijoins (but might be handled by the operator)</p>



<a name="126716671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716671">(May 17 2018 at 20:23)</a>:</h4>
<p>I FIGURED IT OUT!</p>



<a name="126716679"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716679" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716679">(May 17 2018 at 20:23)</a>:</h4>
<p>yes!</p>



<a name="126716730"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716730" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716730">(May 17 2018 at 20:24)</a>:</h4>
<p>So, my antijoin was producing the right tuples. It was just producing them badly. It presumed that if you filter (K,V) then that remains in order, so it doesn't need to re-sort anything. But it forgot about the map that is being applied.</p>



<a name="126716734"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716734" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716734">(May 17 2018 at 20:24)</a>:</h4>
<div class="codehilite"><pre><span></span>Duration { secs: 0, nanos: 213561 } subset is complete: 30
requires, initial size: 3
Duration { secs: 0, nanos: 758060 } requires is complete: 152
Duration { secs: 0, nanos: 821829 } borrow_live_at is complete: 102
</pre></div>



<a name="126716754"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716754" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716754">(May 17 2018 at 20:25)</a>:</h4>
<p>awesome <span class="emoji emoji-1f642" title="simple smile">:simple_smile:</span></p>



<a name="126716809"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716809" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716809">(May 17 2018 at 20:26)</a>:</h4>
<p>pushed</p>



<a name="126716904"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126716904" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126716904">(May 17 2018 at 20:28)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> thanks very much! I'll do the timelyopt conversion tomorrow, I'm really looking forward to seeing those numbers as well</p>



<a name="126717083"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126717083" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126717083">(May 17 2018 at 20:33)</a>:</h4>
<p>No worries <span class="user-mention" data-user-id="116113">@lqd</span>. Fun to do something helpful and relatively understandable.</p>



<a name="126725349"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725349" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725349">(May 18 2018 at 00:31)</a>:</h4>
<p>btw I was poking at your datafrog branch <span class="user-mention" data-user-id="116113">@lqd</span> — seems like we don't need to manually specify the types of all variables, just </p>
<div class="codehilite"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">subset_r1p</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">iteration1</span><span class="p">.</span><span class="n">variable</span><span class="p">(</span><span class="s">&quot;subset_r1p&quot;</span><span class="p">);</span><span class="w"></span>
</pre></div>


<p>will suffice. (I was curious why those types couldn't be inferred. Turns out, they can.)</p>



<a name="126725387"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725387" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725387">(May 18 2018 at 00:32)</a>:</h4>
<p>nice</p>



<a name="126725394"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725394" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725394">(May 18 2018 at 00:32)</a>:</h4>
<p>I found them helpful to debug <span class="emoji emoji-1f642" title="simple smile">:simple_smile:</span> ofc they were initially written by Frank</p>



<a name="126725402"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725402" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725402">(May 18 2018 at 00:33)</a>:</h4>
<p>also helpful for the "mechanized translations" from the datalog rules,  but very verbose indeed</p>



<a name="126725437"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725437" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725437">(May 18 2018 at 00:34)</a>:</h4>
<p>yeah I can imagine they're handy anyway — but if we were going to e.g. generate this code from  a macro, it's nice to not need them.</p>



<a name="126725443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725443">(May 18 2018 at 00:34)</a>:</h4>
<p>absolutely</p>



<a name="126725499"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725499" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725499">(May 18 2018 at 00:36)</a>:</h4>
<p>I'll clean them up tomorrow <span class="emoji emoji-1f642" title="simple smile">:simple_smile:</span></p>



<a name="126725504"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725504" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725504">(May 18 2018 at 00:37)</a>:</h4>
<p>I've finally got a spare moment to read into datafrog</p>



<a name="126725505"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725505" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725505">(May 18 2018 at 00:37)</a>:</h4>
<p>very nice</p>



<a name="126725556"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725556" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725556">(May 18 2018 at 00:39)</a>:</h4>
<p>feels very promising</p>



<a name="126725614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725614">(May 18 2018 at 00:41)</a>:</h4>
<p>also wonder whether rayon could fit in there</p>



<a name="126725620"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725620" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725620">(May 18 2018 at 00:41)</a>:</h4>
<p>if nothing else one could use the <code>par_sort</code> methods</p>



<a name="126725621"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725621" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725621">(May 18 2018 at 00:41)</a>:</h4>
<p>but maybe elsewhere too...</p>



<a name="126725623"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725623" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725623">(May 18 2018 at 00:41)</a>:</h4>
<p>...it's sort of tempting to rewrite to not use <code>Rc&lt;RefCell&lt;..&gt;&gt;</code> etc</p>



<a name="126725624"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725624" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725624">(May 18 2018 at 00:41)</a>:</h4>
<p>i.e., make each <code>Variable</code> an index and have <code>Iteration</code> hold all the data for them</p>



<a name="126725665"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725665" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725665">(May 18 2018 at 00:42)</a>:</h4>
<p>then you would write stuff like <code>iteration.map(target, source, closure)</code> or whatever</p>



<a name="126725666"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725666" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725666">(May 18 2018 at 00:42)</a>:</h4>
<p>instead of <code>target.from_map(source, closure)</code></p>



<a name="126725668"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725668" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725668">(May 18 2018 at 00:42)</a>:</h4>
<p>totally irrelevant of course</p>



<a name="126725670"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725670" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725670">(May 18 2018 at 00:42)</a>:</h4>
<p>though maybe relevant if you were going to use Rayon</p>



<a name="126725671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725671">(May 18 2018 at 00:42)</a>:</h4>
<p>depending</p>



<a name="126725721"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725721" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725721">(May 18 2018 at 00:44)</a>:</h4>
<p>oh interesting</p>



<a name="126725723"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725723" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725723">(May 18 2018 at 00:44)</a>:</h4>
<p>anyway the <em>real</em> question is what will perf look like with the 'timely-opt' variant I guess</p>



<a name="126725726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725726">(May 18 2018 at 00:44)</a>:</h4>
<p>yup</p>



<a name="126725727"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725727" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725727">(May 18 2018 at 00:44)</a>:</h4>
<p>but it seems like the naive was faster</p>



<a name="126725728"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725728" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725728">(May 18 2018 at 00:44)</a>:</h4>
<p>(if I read your msg correctly)</p>



<a name="126725732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725732">(May 18 2018 at 00:44)</a>:</h4>
<p>it was indeed</p>



<a name="126725733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725733">(May 18 2018 at 00:44)</a>:</h4>
<p>though not necessarily faster than w/ <code>distinct_total</code></p>



<a name="126725735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725735">(May 18 2018 at 00:44)</a>:</h4>
<p>(I'd be curious to see that comparison)</p>



<a name="126725738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725738">(May 18 2018 at 00:44)</a>:</h4>
<p>that is, naive w/ distinct_total</p>



<a name="126725739"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725739" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725739">(May 18 2018 at 00:45)</a>:</h4>
<p>I never measured that :)</p>



<a name="126725748"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725748" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725748">(May 18 2018 at 00:45)</a>:</h4>
<p>could be surprising <span class="emoji emoji-1f642" title="simple smile">:simple_smile:</span></p>



<a name="126725750"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725750" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725750">(May 18 2018 at 00:45)</a>:</h4>
<p>one <em>might</em> (I don't know?) expect a similar ratio?</p>



<a name="126725753"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725753" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725753">(May 18 2018 at 00:45)</a>:</h4>
<p>I guess I really have no idea</p>



<a name="126725754"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725754" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725754">(May 18 2018 at 00:45)</a>:</h4>
<p>just gotta measure and see</p>



<a name="126725815"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725815" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725815">(May 18 2018 at 00:46)</a>:</h4>
<p>I remember Jamie's Brandon doing some datafrog work one could say and mentioning exactly that, how important predictable performance was to them</p>



<a name="126725833"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126725833" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126725833">(May 18 2018 at 00:47)</a>:</h4>
<p>I'll try this in the morning</p>



<a name="126726602"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126726602" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126726602">(May 18 2018 at 01:14)</a>:</h4>
<p>Oh just realized that as each variable is of distinct type you can't use indices</p>



<a name="126727288"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126727288" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126727288">(May 18 2018 at 01:40)</a>:</h4>
<p>can't you use a composite index? <code>(type,id)</code></p>



<a name="126736774"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126736774" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126736774">(May 18 2018 at 07:28)</a>:</h4>
<p>quick comments (heading out to the Apple store soon!):</p>
<p>The <code>Rc</code> and <code>RefCell</code>s are optional; they are there to allow a binding of the "read" and "write" halves of each variable, and to allow the <code>Iteration</code> struct to manage all of its variables. If we wanted to break that and insist that users manage both halves and manually called <code>.changed()</code> on each pair, we could avoid them I think. It is perhaps a bit more error-prone, though, in that I could imagine people adding new variables to an existing program and forgetting to step them, which would effectively make them non-recursive (fixed at whatever their initial values are). Could easily complain in <code>Drop</code> if they clearly haven't been stepped completely, but I'm not sure of the right trade-off of "bullet-proofing" vs "simple <code>Send</code> types".</p>



<a name="126737474"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126737474" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126737474">(May 18 2018 at 07:50)</a>:</h4>
<p><span class="user-mention" data-user-id="116106">@Reed Koser</span> thank you that's great!</p>



<a name="126740781"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126740781" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126740781">(May 18 2018 at 09:32)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> re: the <code>Rc</code>, not a big thing. I think the current solution works out quite elegantly.</p>



<a name="126740852"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126740852" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126740852">(May 18 2018 at 09:34)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> you were wondering yesterday, naive DD as is vs naive DD w/ distinct_total comes at 142s vs 118s</p>



<a name="126740863"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126740863" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126740863">(May 18 2018 at 09:34)</a>:</h4>
<p>encouraging</p>



<a name="126740922"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126740922" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126740922">(May 18 2018 at 09:36)</a>:</h4>
<p>naive DF being 52s</p>



<a name="126747926"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126747926" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126747926">(May 18 2018 at 13:02)</a>:</h4>
<p>Can probably speed that up a bit, too. Each of the non-arranged collections (e.g. <code>subset</code>, <code>requires</code>, without a <code>_xyz</code> suffix) can probably be removed and we just use <code>from_map</code> to change one of them to the others.</p>



<a name="126748475"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126748475" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126748475">(May 18 2018 at 13:13)</a>:</h4>
<p>yeah I was also wondering how to use the same indexed inputs between iterations (eg region_live_at and cfg_edge)</p>



<a name="126748582"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126748582" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126748582">(May 18 2018 at 13:16)</a>:</h4>
<p>(I was removing some redundant reindexing between iterations and noticed those)</p>



<a name="126750101"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750101" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750101">(May 18 2018 at 13:40)</a>:</h4>
<p>I was entertaining the idea of making a macro to produce the programs in question</p>



<a name="126750104"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750104" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750104">(May 18 2018 at 13:41)</a>:</h4>
<p>but I decided to "sleep in" instead (until 5:30am :)</p>



<a name="126750114"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750114" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750114">(May 18 2018 at 13:41)</a>:</h4>
<p>kind of glad I didn't though since it sounds like they don't yet reflect "best practice" :)</p>



<a name="126750140"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750140" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750140">(May 18 2018 at 13:41)</a>:</h4>
<p>I don't quite yet see how "arranged" things are reflected..  are those a distinct kind of variable?</p>



<a name="126750338"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750338" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750338">(May 18 2018 at 13:44)</a>:</h4>
<p>I was thinking it was through the "indices" variables (but then again I'm not extremely competent with any of this <span class="emoji emoji-1f921" title="clown face">:clown_face:</span> )</p>



<a name="126750446"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750446" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750446">(May 18 2018 at 13:46)</a>:</h4>
<p>sorry for being slow with the frogopt translation, I'm trying to internalize and come up the mechanized translations rules that would come naturally to the both of you</p>



<a name="126750605"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750605" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750605">(May 18 2018 at 13:49)</a>:</h4>
<blockquote>
<p>I was thinking it was through the "indices" variables (but then again I'm not extremely competent with any of this <span class="emoji emoji-1f921" title="clown face">:clown_face:</span> )</p>
</blockquote>
<p>such modesty :)</p>



<a name="126750613"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750613" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750613">(May 18 2018 at 13:49)</a>:</h4>
<p>I didn't look that closely, apparently, or at least I didn't notice variables named <code>indices</code></p>



<a name="126750615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750615">(May 18 2018 at 13:49)</a>:</h4>
<p>I'll look again later...</p>



<a name="126750657"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750657" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750657">(May 18 2018 at 13:50)</a>:</h4>
<p>I'm mostly hoping you two will just solve all our problems :)</p>



<a name="126750700"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750700" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750700">(May 18 2018 at 13:51)</a>:</h4>
<p>they're not named indices but suffixed with the key</p>



<a name="126750706"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750706" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750706">(May 18 2018 at 13:51)</a>:</h4>
<p>eg subset_r1p</p>



<a name="126750775"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750775" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750775">(May 18 2018 at 13:53)</a>:</h4>
<p>ah those</p>



<a name="126750776"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750776" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750776">(May 18 2018 at 13:53)</a>:</h4>
<p>ok</p>



<a name="126750801"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126750801" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126750801">(May 18 2018 at 13:53)</a>:</h4>
<p>I could be very wrong !!</p>



<a name="126751358"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751358" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751358">(May 18 2018 at 14:04)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> You should be able to pretty cheaply re-use the indices. If you <code>.complete()</code> the variables, you get a sorted <code>Vec&lt;_&gt;</code> out, and if you insert that in a new <code>Variable</code> it should be pretty cheap: the list will be re-sorted, but that should just be a linear scan if the right sort algorithm is used (<code>sort</code> does this, <code>sort_unstable</code> might).</p>



<a name="126751390"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751390" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751390">(May 18 2018 at 14:05)</a>:</h4>
<p>I can demo the "remove <code>subset</code> and just use <code>subset_r1p</code>" thing if you'd like. It's not too magical, and probably just a snippet should make it clear.</p>



<a name="126751393"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751393" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751393">(May 18 2018 at 14:05)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> oh true I didn't think of this, indeed</p>



<a name="126751401"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751401" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751401">(May 18 2018 at 14:05)</a>:</h4>
<p>I think I did this, lemme get you a gist</p>



<a name="126751457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751457">(May 18 2018 at 14:06)</a>:</h4>
<p><a href="https://gist.github.com/lqd/2fbd14646f1404d2222a7d823ed4f1bd" target="_blank" title="https://gist.github.com/lqd/2fbd14646f1404d2222a7d823ed4f1bd">https://gist.github.com/lqd/2fbd14646f1404d2222a7d823ed4f1bd</a></p>



<a name="126751458"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751458" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751458">(May 18 2018 at 14:06)</a>:</h4>
<p>E.g., where right now we have:</p>
<div class="codehilite"><pre><span></span><span class="w">    </span><span class="n">subset_r1p</span><span class="p">.</span><span class="n">from_map</span><span class="p">(</span><span class="o">&amp;</span><span class="n">subset</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">,</span><span class="n">p</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">((</span><span class="n">r1</span><span class="p">,</span><span class="n">p</span><span class="p">),</span><span class="n">r2</span><span class="p">));</span><span class="w"></span>
<span class="w">    </span><span class="n">subset_r2p</span><span class="p">.</span><span class="n">from_map</span><span class="p">(</span><span class="o">&amp;</span><span class="n">subset</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">,</span><span class="n">p</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">((</span><span class="n">r2</span><span class="p">,</span><span class="n">p</span><span class="p">),</span><span class="n">r1</span><span class="p">));</span><span class="w"></span>
<span class="w">    </span><span class="n">subset_p</span><span class="p">.</span><span class="n">from_map</span><span class="p">(</span><span class="o">&amp;</span><span class="n">subset</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">,</span><span class="n">p</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">p</span><span class="p">,(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">)));</span><span class="w"></span>
</pre></div>


<p>we can just skip <code>subset</code> and do</p>
<div class="codehilite"><pre><span></span><span class="w">    </span><span class="c1">// subset_r1p is now the thing we build up</span>
<span class="w">    </span><span class="n">subset_r2p</span><span class="p">.</span><span class="n">from_map</span><span class="p">(</span><span class="o">&amp;</span><span class="n">subset_r1p</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">((</span><span class="n">r1</span><span class="p">,</span><span class="n">p</span><span class="p">),</span><span class="n">r2</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">((</span><span class="n">r2</span><span class="p">,</span><span class="n">p</span><span class="p">),</span><span class="n">r1</span><span class="p">));</span><span class="w"></span>
<span class="w">    </span><span class="n">subset_p</span><span class="p">.</span><span class="n">from_map</span><span class="p">(</span><span class="o">&amp;</span><span class="n">subset_r1p</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">((</span><span class="n">r1</span><span class="p">,</span><span class="n">p</span><span class="p">),</span><span class="n">r2</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">p</span><span class="p">,(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">)));</span><span class="w"></span>
</pre></div>



<a name="126751492"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751492" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751492">(May 18 2018 at 14:07)</a>:</h4>
<p>oh no so it's not what I was talking about, (and I think you actually already did this) using the indexed relations from iteration to the next when possible</p>



<a name="126751553"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751553" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751553">(May 18 2018 at 14:08)</a>:</h4>
<p>very nice, thank you. I'll try to see if I can add to frogopt when I've progressed enough</p>



<a name="126751561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751561">(May 18 2018 at 14:09)</a>:</h4>
<p>Sorry, two things here:</p>
<p>1. Capturing <code>subset_r1p</code> from the first iteration (subset) and using it in the second iteration (requires). You are doing this in the fragment, which is great.<br>
2. Not having a <code>subset</code> or a <code>requires</code>, and only keeping indexed lists.</p>



<a name="126751664"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751664" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751664">(May 18 2018 at 14:11)</a>:</h4>
<p>this will help as well indeed, they are pretty big on clap</p>



<a name="126751672"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126751672" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126751672">(May 18 2018 at 14:11)</a>:</h4>
<p>I like that there's a lot of potential to improve</p>



<a name="126752344"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126752344" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126752344">(May 18 2018 at 14:25)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> such indexed lists would be where in DD we'd use "arranged" lists right ?</p>



<a name="126752351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126752351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126752351">(May 18 2018 at 14:25)</a>:</h4>
<p>Erm, sure, except so much simpler here.</p>



<a name="126752398"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126752398" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126752398">(May 18 2018 at 14:26)</a>:</h4>
<p>A <code>Relation</code>'s <code>into</code> method sorts and deduplicates, which isn't free, but if the list is already sorted it is really. cheap.</p>



<a name="126752459"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126752459" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126752459">(May 18 2018 at 14:27)</a>:</h4>
<p>The spirit is the same (cheap to re-use) but operationally they are pretty different. The arranged re-use is more like when we use a <code>Variable</code> multiple times in the same query.</p>



<a name="126752521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126752521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126752521">(May 18 2018 at 14:28)</a>:</h4>
<p><span class="emoji emoji-1f44d" title="thumbs up">:thumbs_up:</span></p>



<a name="126752697"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126752697" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126752697">(May 18 2018 at 14:33)</a>:</h4>
<p>Btw, if you want any help at any point just holler. I'm done with my work for the week and have power again. :)  Resting for a bit before heading out to explore Vienna.</p>



<a name="126753157"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753157" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753157">(May 18 2018 at 14:43)</a>:</h4>
<p>I'm just at the point I think I've translated the 1st join lol</p>



<a name="126753216"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753216" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753216">(May 18 2018 at 14:44)</a>:</h4>
<p>I was a bit unsure what the granularity was to cut the queries into different Iterations</p>



<a name="126753217"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753217" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753217">(May 18 2018 at 14:44)</a>:</h4>
<p>(and I had the same problem knowing when to use scopes with DD)</p>



<a name="126753324"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753324" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753324">(May 18 2018 at 14:47)</a>:</h4>
<p>esp since this timelyopt conversion has a more complicated graph than the naive one (and which you did 99% of)</p>



<a name="126753655"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753655" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753655">(May 18 2018 at 14:55)</a>:</h4>
<p>I think in principle you can just put everything in one iteration, if you want.</p>



<a name="126753709"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753709" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753709">(May 18 2018 at 14:56)</a>:</h4>
<p>The only reason you "need" multiple ones is when you want to use antijoin. Otherwise, all of the relations contribute to each other positively, and can just co-develop. Nothing especially good happens by completing one of them first and then doing the next (perhaps lower peak memory utilization?).</p>



<a name="126753738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753738">(May 18 2018 at 14:57)</a>:</h4>
<blockquote>
<p>when you want to use antijoin.</p>
</blockquote>
<p>meaning, when you want to derive something iteratively, and then use it as the right-hand side of an antijoin.</p>



<a name="126753795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753795">(May 18 2018 at 14:58)</a>:</h4>
<p>yeah I was gonna say that <em>our</em> use of antijoin should be fine</p>



<a name="126753798"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753798" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753798">(May 18 2018 at 14:58)</a>:</h4>
<p>with one <del>iteration</del> loop</p>



<a name="126753893"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753893" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753893">(May 18 2018 at 15:00)</a>:</h4>
<p>oh very interesting, and rn we're only doing antijoins of inputs</p>



<a name="126753916"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753916" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753916">(May 18 2018 at 15:01)</a>:</h4>
<p>Right, all the rules are monotonic (positively so) with respect to all variables other than <code>killed</code>, which is static over the rounds of iteration.</p>



<a name="126753918"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126753918" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126753918">(May 18 2018 at 15:01)</a>:</h4>
<p>right. usually datalog people talk about "stratifying" the programs in layers here when it comes to negation, basically — that is, if you do an antijoin of a non-input relation R, you have to have  a "layer" such that R and all of its dependencies are completed first</p>



<a name="126754063"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754063" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754063">(May 18 2018 at 15:04)</a>:</h4>
<p>thanks to the both of you :)</p>



<a name="126754080"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754080" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754080">(May 18 2018 at 15:05)</a>:</h4>
<p>btw <span class="user-mention" data-user-id="116609">@Frank McSherry</span> have you seen a sketch of what the "timelyopt" dl looked like ? something like <a href="https://gist.github.com/lqd/88f6eb5dd75e3850d2abc1242dbcea4a" target="_blank" title="https://gist.github.com/lqd/88f6eb5dd75e3850d2abc1242dbcea4a">https://gist.github.com/lqd/88f6eb5dd75e3850d2abc1242dbcea4a</a> (some things are missing/different but giving the same answers on the specific test datasets)</p>



<a name="126754268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754268">(May 18 2018 at 15:09)</a>:</h4>
<p>I have seen that; I think <span class="user-mention" data-user-id="116009">@nikomatsakis</span> has an even newer version though, right? E.g. stuff like </p>
<div class="codehilite"><pre><span></span>    // dead_region_requires(R, B, P, Q) :-
    //   requires(R, B, P),
    //   !killed(B, P),
    //   cfg_edge(P, Q),
    //   !region_live_at(R, Q).
</pre></div>



<a name="126754281"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754281" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754281">(May 18 2018 at 15:09)</a>:</h4>
<p>I tihnk it's the same except universal_regions</p>



<a name="126754336"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754336" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754336">(May 18 2018 at 15:10)</a>:</h4>
<p>I'm up for implementing any random program, but don't want to grab that away from you. :)</p>



<a name="126754378"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754378" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754378">(May 18 2018 at 15:11)</a>:</h4>
<p>Yeah current <code>timely_opt.rs</code> has all sorts of heinous subset/requires co-development, and lots of <code>dead_can_reach_live</code> horror story plotlines going on in it.</p>



<a name="126754379"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754379" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754379">(May 18 2018 at 15:11)</a>:</h4>
<p>(eg in soufflé I wasn't easily able to have facts containing the universal region points, so I used the rules until then; not a problem with DD or data-<span class="emoji emoji-1f438" title="frog">:frog:</span>)</p>



<a name="126754410"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754410" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754410">(May 18 2018 at 15:11)</a>:</h4>
<p>I believe that Polonius master has the latest version of the timely-opt computation</p>



<a name="126754462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754462">(May 18 2018 at 15:12)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span>  I'd gladly take help but won't take you away from visiting Vienna</p>



<a name="126754464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754464">(May 18 2018 at 15:12)</a>:</h4>
<blockquote>
<p>Yeah current <code>timely_opt.rs</code> has all sorts of heinous subset/requires co-development, and lots of <code>dead_can_reach_live</code> horror story plotlines going on in it.</p>
</blockquote>
<p>( I think this can all be one big iteration though )</p>



<a name="126754475"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754475" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754475">(May 18 2018 at 15:12)</a>:</h4>
<p>oh I didn't see the latest revision then :/</p>



<a name="126754500"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754500" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754500">(May 18 2018 at 15:13)</a>:</h4>
<p>this is me looking at Niko's "detransitize" branch; maybe not the current one (no clue what "polonius" is; some sort of Bond villian org codename probably).</p>



<a name="126754506"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754506" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754506">(May 18 2018 at 15:13)</a>:</h4>
<p><a href="https://github.com/rust-lang-nursery/polonius/" target="_blank" title="https://github.com/rust-lang-nursery/polonius/">https://github.com/rust-lang-nursery/polonius/</a></p>



<a name="126754511"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754511" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754511">(May 18 2018 at 15:13)</a>:</h4>
<p>it's the new name for the borrow-check repo :)</p>



<a name="126754558"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754558" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754558">(May 18 2018 at 15:14)</a>:</h4>
<p>Now! With extra <del>fun</del> Shakespeare references!</p>



<a name="126754566"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754566" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754566">(May 18 2018 at 15:14)</a>:</h4>
<p>anyway, <a href="https://github.com/rust-lang-nursery/polonius/blob/master/src/output/timely_opt.rs" target="_blank" title="https://github.com/rust-lang-nursery/polonius/blob/master/src/output/timely_opt.rs">this is the most up to date</a></p>



<a name="126754567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754567">(May 18 2018 at 15:14)</a>:</h4>
<p>it is probably the same as the detransitize branch</p>



<a name="126754653"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754653" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754653">(May 18 2018 at 15:16)</a>:</h4>
<p>Okies, I may go and try to hack on that for a bit. Maybe grab a glass of wein on the Danube (in Wien) and type a bit taking in the evening.</p>



<a name="126754753"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754753" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754753">(May 18 2018 at 15:18)</a>:</h4>
<p>I myself am verifying I correctly did like the 3rd clause ... <code>live_to_dead_regions</code></p>



<a name="126754766"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754766" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754766">(May 18 2018 at 15:18)</a>:</h4>
<p>/me is having a lot of fun watching this unfold</p>



<a name="126754769"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754769" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754769">(May 18 2018 at 15:18)</a>:</h4>
<p>(slowly getting the hang of it)</p>



<a name="126754785"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754785" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754785">(May 18 2018 at 15:19)</a>:</h4>
<p>no pressure :p</p>



<a name="126754789"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754789" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754789">(May 18 2018 at 15:19)</a>:</h4>
<p>![popcorn](<a href="http://i0.kym-cdn.com/photos/images/newsfeed/000/895/845/2f9.jpg" target="_blank" title="http://i0.kym-cdn.com/photos/images/newsfeed/000/895/845/2f9.jpg">http://i0.kym-cdn.com/photos/images/newsfeed/000/895/845/2f9.jpg</a>)</p>
<div class="message_inline_image"><a href="http://i0.kym-cdn.com/photos/images/newsfeed/000/895/845/2f9.jpg" target="_blank" title="http://i0.kym-cdn.com/photos/images/newsfeed/000/895/845/2f9.jpg"><img src="https://uploads.zulipusercontent.net/4eed7fdd0ca614b1f16c7f425fe2c1353a32ab6e/687474703a2f2f69302e6b796d2d63646e2e636f6d2f70686f746f732f696d616765732f6e657773666565642f3030302f3839352f3834352f3266392e6a7067"></a></div>



<a name="126754951"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126754951" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126754951">(May 18 2018 at 15:23)</a>:</h4>
<p>yesss, 1 done, 1 million to go</p>



<a name="126755122"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126755122" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126755122">(May 18 2018 at 15:26)</a>:</h4>
<p>Random observation from before, probably still applies: in the clap example there were several thousand rounds of derivations, which makes me think there is <em>lots</em> of cfg crawling going on. If there is still a plan to collapse down indistinguishable regions, that could probably still have a positive effect!</p>



<a name="126755139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126755139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126755139">(May 18 2018 at 15:26)</a>:</h4>
<p>there are plans indeed :D</p>



<a name="126755156"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126755156" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126755156">(May 18 2018 at 15:27)</a>:</h4>
<p><a href="https://github.com/rust-lang-nursery/polonius/issues/20" target="_blank" title="https://github.com/rust-lang-nursery/polonius/issues/20">https://github.com/rust-lang-nursery/polonius/issues/20</a></p>



<a name="126755232"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126755232" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126755232">(May 18 2018 at 15:28)</a>:</h4>
<p>Neat! We can probably do it in Datalog too, fwiw (connected components of edges are indistinguishable if their application is unimpeded by any of the <code>region_live_at</code> transitions, and <code>killed</code>, I think).</p>



<a name="126755260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126755260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126755260">(May 18 2018 at 15:29)</a>:</h4>
<p>sneaking off for a bit; will report back later!</p>



<a name="126756024"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756024" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756024">(May 18 2018 at 15:48)</a>:</h4>
<blockquote>
<p>Neat! We can probably do it in Datalog too, fwiw (connected components of edges are indistinguishable if their application is unimpeded by any of the <code>region_live_at</code> transitions, and <code>killed</code>, I think).</p>
</blockquote>
<p>that was my plan</p>



<a name="126756032"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756032" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756032">(May 18 2018 at 15:48)</a>:</h4>
<p>but we don't have all the pieces we need for that yet</p>



<a name="126756040"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756040" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756040">(May 18 2018 at 15:48)</a>:</h4>
<p>almost</p>



<a name="126756044"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756044" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756044">(May 18 2018 at 15:48)</a>:</h4>
<p>anyway, I'd like to keep that in our back pocket :)</p>



<a name="126756050"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756050" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756050">(May 18 2018 at 15:48)</a>:</h4>
<p>it seems like we've got a decent chance of getting the perf we need without</p>



<a name="126756054"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756054" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756054">(May 18 2018 at 15:48)</a>:</h4>
<p>in which case it's just gravy :)</p>



<a name="126756322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756322">(May 18 2018 at 15:54)</a>:</h4>
<p>&lt;insert homer mmm gravy meme here&gt;</p>



<a name="126756415"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756415" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756415">(May 18 2018 at 15:57)</a>:</h4>
<p>One thing I don't currently like about datafrog is that the variable declarations happen so far from their use. Hard to write rules without lots of scrolling around. =/</p>



<a name="126756464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756464">(May 18 2018 at 15:58)</a>:</h4>
<p>Btw, sitting on the river, having some dark czech beer, enjoying the weather. just fyi about life choices.</p>



<a name="126756978"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756978" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756978">(May 18 2018 at 16:09)</a>:</h4>
<p>I'm writing each mecanized step as well so I have 100 lines between one operator call and the var decl :D</p>



<a name="126756989"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126756989" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126756989">(May 18 2018 at 16:09)</a>:</h4>
<p>fwiw I'm in sunny south of france frank :p</p>



<a name="126757078"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757078" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757078">(May 18 2018 at 16:10)</a>:</h4>
<p>(not far from where Prolog was invented)</p>



<a name="126757137"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757137" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757137">(May 18 2018 at 16:12)</a>:</h4>
<p>How's boston, <span class="user-mention" data-user-id="116009">@nikomatsakis</span> ?</p>



<a name="126757151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757151">(May 18 2018 at 16:13)</a>:</h4>
<p>/me can't complain about sunny Florida</p>



<a name="126757214"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757214" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757214">(May 18 2018 at 16:14)</a>:</h4>
<p>3/4 :D</p>



<a name="126757244"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757244" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757244">(May 18 2018 at 16:15)</a>:</h4>
<p>:D</p>



<a name="126757508"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757508" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757508">(May 18 2018 at 16:21)</a>:</h4>
<p>ok, polonius version "code complete"!</p>



<a name="126757535"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757535" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757535">(May 18 2018 at 16:21)</a>:</h4>
<p>(meaning: typed a lot, now time to see what happens when I <code>cargo check</code> it.)</p>



<a name="126757606"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757606" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757606">(May 18 2018 at 16:23)</a>:</h4>
<p>I had barely finished <code>dead_region_requires</code> ...</p>



<a name="126757763"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757763" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757763">(May 18 2018 at 16:26)</a>:</h4>
<p>I'm trying out a weird programming style. Write all the rules w/o declaring anything, see what you need, then go and define the things.</p>



<a name="126757787"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757787" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757787">(May 18 2018 at 16:27)</a>:</h4>
<p>while I spend 90% of my time scrolling</p>



<a name="126757850"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757850" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757850">(May 18 2018 at 16:29)</a>:</h4>
<p>Yeah, I did that for a while and then thought "I have so much beer in my belly, why don't I just write the program first will surely work out".</p>



<a name="126757935"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126757935" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126757935">(May 18 2018 at 16:31)</a>:</h4>
<p><span class="emoji emoji-1f37a" title="beer">:beer:</span> -driven development</p>



<a name="126758329"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758329" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758329">(May 18 2018 at 16:40)</a>:</h4>
<p>Rapidly approaching Ballmer peak.</p>



<a name="126758414"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758414" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758414">(May 18 2018 at 16:42)</a>:</h4>
<p>it's running...</p>



<a name="126758424"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758424" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758424">(May 18 2018 at 16:43)</a>:</h4>
<p>super fast.</p>



<a name="126758425"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758425" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758425">(May 18 2018 at 16:43)</a>:</h4>
<p>awesome</p>



<a name="126758428"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758428" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758428">(May 18 2018 at 16:43)</a>:</h4>
<div class="codehilite"><pre><span></span>     Running `target/release/borrow-check -a Datafrog &#39;inputs/clap-rs/app-parser-{{impl}}-add_defaults&#39; --skip-tuples`
Duration { secs: 0, nanos: 523194318 }  borrow_live_at is complete: 0
--------------------------------------------------
Directory: inputs/clap-rs/app-parser-{{impl}}-add_defaults
Time: 0.549s
Echidnatron%
</pre></div>



<a name="126758431"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758431" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758431">(May 18 2018 at 16:43)</a>:</h4>
<p>sec.</p>



<a name="126758496"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758496" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758496">(May 18 2018 at 16:44)</a>:</h4>
<p>:)</p>



<a name="126758519"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758519" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758519">(May 18 2018 at 16:45)</a>:</h4>
<p>debugging this is going to be rough. I forgot to load <code>requires</code> with <code>borrow_region</code>, but still not right yet:</p>
<div class="codehilite"><pre><span></span>     Running `target/release/borrow-check -a Datafrog &#39;inputs/clap-rs/app-parser-{{impl}}-add_defaults&#39; --skip-tuples`
Duration { secs: 2, nanos: 122990259 }  borrow_live_at is complete: 391740
--------------------------------------------------
Directory: inputs/clap-rs/app-parser-{{impl}}-add_defaults
Time: 2.174s
Echidnatron%
</pre></div>



<a name="126758566"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758566" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758566">(May 18 2018 at 16:46)</a>:</h4>
<p>only off by about 2x, right? &gt;.&gt;</p>



<a name="126758575"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758575" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758575">(May 18 2018 at 16:47)</a>:</h4>
<p>:D</p>



<a name="126758583"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758583" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758583">(May 18 2018 at 16:47)</a>:</h4>
<p>yeah debugging was a bit rough for me even on the simpler naive program</p>



<a name="126758587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758587">(May 18 2018 at 16:47)</a>:</h4>
<p>that's why I went step by step comparing to DD, but that requires a 2.5mins compilation per clause ....</p>



<a name="126758646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758646">(May 18 2018 at 16:49)</a>:</h4>
<div class="codehilite"><pre><span></span>warning: unused variable: `dead_region_requires`
  --&gt; src/output/datafrog.rs:47:13
   |
47 |         let dead_region_requires = iteration.variable::&lt;(Region, Loan, Point, Point)&gt;(&quot;dead_region_requires&quot;);
   |             ^^^^^^^^^^^^^^^^^^^^ help: consider using `_dead_region_requires` instead
</pre></div>



<a name="126758647"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758647" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758647">(May 18 2018 at 16:49)</a>:</h4>
<p>Hint #1</p>



<a name="126758648"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758648" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758648">(May 18 2018 at 16:49)</a>:</h4>
<p>(mostly to side step ungrounded vars w/ soufflé)</p>



<a name="126758652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758652">(May 18 2018 at 16:49)</a>:</h4>
<p>this compiler is helpful!</p>



<a name="126758729"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758729" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758729">(May 18 2018 at 16:51)</a>:</h4>
<p>I'm coming up to <code>dead_can_reach</code> -- I'm loving theses clauses btw, it's like a Walking Dead homage, dead_can_reach_origins the prequel, dead_can_reach_live the Broadway musical</p>



<a name="126758774"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758774" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758774">(May 18 2018 at 16:52)</a>:</h4>
<p>fwiw: <a href="https://gist.github.com/frankmcsherry/adb9ed3c433eb9f4ddaf591ad9888dea" target="_blank" title="https://gist.github.com/frankmcsherry/adb9ed3c433eb9f4ddaf591ad9888dea">https://gist.github.com/frankmcsherry/adb9ed3c433eb9f4ddaf591ad9888dea</a></p>



<a name="126758776"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758776" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758776">(May 18 2018 at 16:52)</a>:</h4>
<p>I had it as "dead can reach live? oh no! run live, run!"</p>



<a name="126758846"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758846" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758846">(May 18 2018 at 16:55)</a>:</h4>
<p>mine is 100 comments per join <span class="emoji emoji-1f62d" title="sob">:sob:</span> <a href="https://gist.github.com/lqd/e6bcc6bde2b0fe9a14e3f2fa1b57904c" target="_blank" title="https://gist.github.com/lqd/e6bcc6bde2b0fe9a14e3f2fa1b57904c">https://gist.github.com/lqd/e6bcc6bde2b0fe9a14e3f2fa1b57904c</a></p>



<a name="126758868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758868">(May 18 2018 at 16:55)</a>:</h4>
<p>One thing I'm not sure of is whether I grokked Niko's DD code. Embarrassingly, it is apparently hard to read... :)</p>



<a name="126758929"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758929" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758929">(May 18 2018 at 16:56)</a>:</h4>
<p>tbf the beers might be part of the reason why :D</p>



<a name="126758942"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126758942" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126758942">(May 18 2018 at 16:57)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> beautiful right now :) springtime is the best ...</p>



<a name="126759050"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759050" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759050">(May 18 2018 at 16:59)</a>:</h4>
<blockquote>
<p>One thing I'm not sure of is whether I grokked Niko's DD code. Embarrassingly, it is apparently hard to read... :)</p>
</blockquote>
<p>uh oh =)</p>



<a name="126759105"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759105" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759105">(May 18 2018 at 17:00)</a>:</h4>
<p>I tried to keep datalog comments because — I confess —  find the DD by itself kind of "write only" (but quite mechanical and easy to write once you have datalog)</p>



<a name="126759118"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759118" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759118">(May 18 2018 at 17:00)</a>:</h4>
<p>maybe that's not enough though</p>



<a name="126759121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759121">(May 18 2018 at 17:00)</a>:</h4>
<p>(Frank BTW, any time it's not fun debugging anymore vs enjoying Vienna, I can also "help")</p>



<a name="126759124"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759124" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759124">(May 18 2018 at 17:00)</a>:</h4>
<p>Well, all the beer is drunk, so I'm afraid I must go.</p>



<a name="126759133"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759133" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759133">(May 18 2018 at 17:01)</a>:</h4>
<p>I'll just leave this:</p>
<div class="codehilite"><pre><span></span>Running `target/release/borrow-check -a Datafrog &#39;inputs/clap-rs/app-parser-{{impl}}-add_defaults&#39; --skip-tuples`
Duration { secs: 16, nanos: 743834090 } borrow_live_at is complete: 832392
--------------------------------------------------
Directory: inputs/clap-rs/app-parser-{{impl}}-add_defaults
Time: 16.839s
Echidnatron%
</pre></div>



<a name="126759139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759139">(May 18 2018 at 17:01)</a>:</h4>
<p>he's already debugged it</p>



<a name="126759143"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759143" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759143">(May 18 2018 at 17:01)</a>:</h4>
<p>is that the right number of tuples? :)</p>



<a name="126759144"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759144" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759144">(May 18 2018 at 17:01)</a>:</h4>
<p>yeah</p>



<a name="126759146"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759146" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759146">(May 18 2018 at 17:02)</a>:</h4>
<p>seems to be approx the same time, which is interesting</p>



<a name="126759186"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759186" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759186">(May 18 2018 at 17:02)</a>:</h4>
<p>oh well I guess</p>



<a name="126759187"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759187" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759187">(May 18 2018 at 17:02)</a>:</h4>
<p>I am comparing two different computers ;)</p>



<a name="126759311"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759311" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759311">(May 18 2018 at 17:05)</a>:</h4>
<p>we might need the cfg compression after all, fiddle with different sorting methods (even without rayon's par sort), try and reduce alloc and temporaries</p>



<a name="126759375"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759375" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759375">(May 18 2018 at 17:06)</a>:</h4>
<p>I'm not worried yet :)</p>



<a name="126759379"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759379" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759379">(May 18 2018 at 17:06)</a>:</h4>
<p>between CFG compression + the location-insensitive-pre-filter I think we'll make big wins</p>



<a name="126759458"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759458" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759458">(May 18 2018 at 17:08)</a>:</h4>
<p>even if it had exactly DD like performance (not thinking about ease of writing, which we could macro) it might be still worth it for the compile time win, easier maintenance, good optimization potential, etc</p>



<a name="126759555"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759555" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759555">(May 18 2018 at 17:11)</a>:</h4>
<p>oh, definitely!</p>



<a name="126759562"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759562" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759562">(May 18 2018 at 17:11)</a>:</h4>
<p>this seems like a much easier story re: integration</p>



<a name="126759573"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759573" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759573">(May 18 2018 at 17:11)</a>:</h4>
<p>(actually, <span class="user-mention" data-user-id="116083">@pnkfelix</span>, I wonder if it's worth waiting for the datafrog story to play out — that might completely obviate the need to integrate an executable into rustc)</p>



<a name="126759686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759686">(May 18 2018 at 17:14)</a>:</h4>
<p>(I'll also convert the pre-pass to datafrog, at a snail's pace evidently)</p>



<a name="126759692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759692">(May 18 2018 at 17:14)</a>:</h4>
<p>Sorry! In actual fact it started raining and the beer-man decided to close up shop, meaning I had to boogie. Obvs not all the beer had been drunk (it's Austria!).</p>



<a name="126759696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759696">(May 18 2018 at 17:14)</a>:</h4>
<p>Gist updated with the correct version.</p>



<a name="126759769"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759769" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759769">(May 18 2018 at 17:16)</a>:</h4>
<p>Full detail:</p>
<div class="codehilite"><pre><span></span>    Finished release [optimized] target(s) in 18.62s
     Running `target/release/borrow-check -a Datafrog &#39;inputs/clap-rs/app-parser-{{impl}}-add_defaults&#39; --skip-tuples`
Duration { secs: 16, nanos: 548798709 } borrow_live_at is complete: 832392
--------------------------------------------------
Directory: inputs/clap-rs/app-parser-{{impl}}-add_defaults
Time: 16.638s
Echidnatron%
</pre></div>


<p>means it now runs faster than it takes to compile. Ball is in your court, <span class="user-mention" data-user-id="116009">@nikomatsakis</span>  ;)</p>



<a name="126759860"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759860" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759860">(May 18 2018 at 17:19)</a>:</h4>
<p>Looks like (profiling) almost all the time is spent in deduplication. Probably what's going on is that there are relatively fewer <code>distinct_total</code> calls in the differential code, whereas datafrog is doing a distinct for every variable (incl intermediates, etc).</p>



<a name="126759931"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759931" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759931">(May 18 2018 at 17:20)</a>:</h4>
<p>that makes sense; I did spend some time thinking about where duplicates could be introduced</p>



<a name="126759934"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759934" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759934">(May 18 2018 at 17:20)</a>:</h4>
<p>I suppose we could make that explicit in datafrog?</p>



<a name="126759948"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759948" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759948">(May 18 2018 at 17:21)</a>:</h4>
<p>ps I really like the model of "manually" compiling to a nice set of primitives</p>



<a name="126759952"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759952" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759952">(May 18 2018 at 17:21)</a>:</h4>
<p>that is, I like us having the ability to map it back to datalog, but also not being reliant on some automated optimizer</p>



<a name="126759954"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759954" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759954">(May 18 2018 at 17:21)</a>:</h4>
<p>for perf reasons probably makes sense. could probably just have a bool with each variable that is "do distinct", at which point it is on the programmer to make sure they have distincts set up to converge properly (no cycles w/o distincts).</p>



<a name="126759957"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126759957" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126759957">(May 18 2018 at 17:21)</a>:</h4>
<p>(to apply domain knowledge of this kind)</p>



<a name="126760090"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760090" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760090">(May 18 2018 at 17:24)</a>:</h4>
<p>on my machine, timely 11s vs <span class="emoji emoji-1f438" title="frog">:frog:</span> 15s -- manual distincts would be <span class="emoji emoji-1f44d" title="thumbs up">:thumbs_up:</span></p>



<a name="126760133"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760133" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760133">(May 18 2018 at 17:25)</a>:</h4>
<p>Manual distincts:</p>
<div class="codehilite"><pre><span></span>Running `target/release/borrow-check -a Datafrog &#39;inputs/clap-rs/app-parser-{{impl}}-add_defaults&#39; --skip-tuples`
Duration { secs: 9, nanos: 640609767 }  borrow_live_at is complete: 832392
--------------------------------------------------
Directory: inputs/clap-rs/app-parser-{{impl}}-add_defaults
Time: 9.729s
Echidnatron%
</pre></div>



<a name="126760180"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760180" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760180">(May 18 2018 at 17:26)</a>:</h4>
<blockquote>
<p>on my machine, timely 11s vs <span class="emoji emoji-1f438" title="frog">:frog:</span> 15s -- manual distincts would be <span class="emoji emoji-1f44d" title="thumbs up">:thumbs_up:</span></p>
</blockquote>
<p>I think you mean: <span class="emoji emoji-1f550" title="clock1">:clock1:</span> 11s vs <span class="emoji emoji-1f438" title="frog">:frog:</span> 15s</p>



<a name="126760208"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760208" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760208">(May 18 2018 at 17:27)</a>:</h4>
<p>also i confess I find data-<span class="emoji emoji-1f438" title="frog">:frog:</span> hard to read and prefer just <span class="emoji emoji-1f438" title="frog">:frog:</span></p>



<a name="126760219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760219">(May 18 2018 at 17:27)</a>:</h4>
<p>emoji-word hybrids blow my mind</p>



<a name="126760223"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760223" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760223">(May 18 2018 at 17:27)</a>:</h4>
<p>Now <em>that</em> is the nit to end all nits.</p>



<a name="126760292"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760292" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760292">(May 18 2018 at 17:29)</a>:</h4>
<p>(if you'll allow me, I just want to say that Niko, Frank -- and also Jamie who's not doing Rust anymore -- are 3 of my favorite technical writers/people/etc in the world, so I'm extremely happy rn)</p>



<a name="126760297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760297">(May 18 2018 at 17:29)</a>:</h4>
<p>Jamie is still doing Rust I think...</p>



<a name="126760301"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760301" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760301">(May 18 2018 at 17:29)</a>:</h4>
<p>Just, not as publicly. &gt;.&gt;</p>



<a name="126760348"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760348" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760348">(May 18 2018 at 17:30)</a>:</h4>
<p>:) very Julia rn, think he did a talk extremely recently on Imp</p>



<a name="126760433"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760433" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760433">(May 18 2018 at 17:32)</a>:</h4>
<p>compilation staging seemed key to this work</p>



<a name="126760448"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760448" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760448">(May 18 2018 at 17:33)</a>:</h4>
<p>Profiling on the manual distinct runs, btw:</p>



<a name="126760452"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760452" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760452">(May 18 2018 at 17:33)</a>:</h4>
<p><a href="/user_uploads/4715/_RxOBxhoLOTBbtg6wo9lmOuv/Screen-Shot-2018-05-18-at-19.32.49.png" target="_blank" title="Screen-Shot-2018-05-18-at-19.32.49.png">Screen-Shot-2018-05-18-at-19.32.49.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/_RxOBxhoLOTBbtg6wo9lmOuv/Screen-Shot-2018-05-18-at-19.32.49.png" target="_blank" title="Screen-Shot-2018-05-18-at-19.32.49.png"><img src="/user_uploads/4715/_RxOBxhoLOTBbtg6wo9lmOuv/Screen-Shot-2018-05-18-at-19.32.49.png"></a></div>



<a name="126760462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760462">(May 18 2018 at 17:33)</a>:</h4>
<p>I don't know how to interpret that :)</p>



<a name="126760469"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760469" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760469">(May 18 2018 at 17:33)</a>:</h4>
<p>except that "shuffling data" is a big part...</p>



<a name="126760513"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760513" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760513">(May 18 2018 at 17:34)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> do you think we could process some of the tuples creation for each loop iteration, in parallel (and if it would help) as we have the base relations and the deltas and can join them I think independently + union the 3 subresults</p>



<a name="126760516"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760516" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760516">(May 18 2018 at 17:34)</a>:</h4>
<p>The sorting and such does a bit of memmove action. I can probably tidy some of it up, but it's probably somewhat intrinsic to the approach.</p>



<a name="126760531"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760531" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760531">(May 18 2018 at 17:35)</a>:</h4>
<p>Yes, that could totally work. All of the update steps do read-only access to their inputs, populate a vec, and then append that vec to a "todo list" for each variable.</p>



<a name="126760542"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760542" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760542">(May 18 2018 at 17:35)</a>:</h4>
<p>(and rayon's par_iter and par_sort)</p>



<a name="126760543"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760543" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760543">(May 18 2018 at 17:35)</a>:</h4>
<p>Although, looking at the profiling, not so much time is spend in <code>join_into</code> and <code>antijoin_into</code>, so perhaps it would be better to parallelize the remaining distincts.</p>



<a name="126760628"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760628" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760628">(May 18 2018 at 17:37)</a>:</h4>
<p>I'm sure there are plenty more optimizations too. Each of the named relations is in un-indexed form, and then indexed a few different ways. That should mean we can remove at least one instance of each named relation, which should do something positive probably. &gt;.&gt;</p>



<a name="126760685"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760685" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760685">(May 18 2018 at 17:38)</a>:</h4>
<p>Random other things too: the way I merge two sorted lists is to append one on to the other and call <code>sort()</code>, which .. you might have opinions about.</p>



<a name="126760693"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760693" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760693">(May 18 2018 at 17:38)</a>:</h4>
<p>what is the current <span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span> vs <span class="emoji emoji-1f438" title="frog">:frog:</span> results?</p>



<a name="126760701"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760701" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760701">(May 18 2018 at 17:38)</a>:</h4>
<p>with 'less dedup' I mean</p>



<a name="126760732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760732">(May 18 2018 at 17:39)</a>:</h4>
<p>(in my recent tests of your previous ideas removing some of those bigger relations resulted in visible wins, albeit on the Naive prog / <span class="emoji emoji-1f438" title="frog">:frog:</span>)</p>



<a name="126760763"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760763" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760763">(May 18 2018 at 17:39)</a>:</h4>
<p>It's a bit hard to say; lemme push the manual distinct repo, and then gist my code.</p>



<a name="126760847"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760847" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760847">(May 18 2018 at 17:41)</a>:</h4>
<p>Manual distinct pushed (new method: <code>variable_indistinct()</code>) and gist updated with code that uses it.</p>



<a name="126760908"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760908" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760908">(May 18 2018 at 17:42)</a>:</h4>
<p>But, perf seems good enough that next step is probably collapsing CFG. ;)</p>



<a name="126760940"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760940" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760940">(May 18 2018 at 17:43)</a>:</h4>
<p><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span> 11s vs <span class="emoji emoji-1f438" title="frog">:frog:</span> 8s</p>



<a name="126760953"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760953" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760953">(May 18 2018 at 17:43)</a>:</h4>
<p>(on this beefy machine)</p>



<a name="126760972"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126760972" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126760972">(May 18 2018 at 17:43)</a>:</h4>
<p>yeah, this is great!</p>



<a name="126761028"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761028" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761028">(May 18 2018 at 17:44)</a>:</h4>
<p>25% win, rounding down a bit</p>



<a name="126761034"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761034" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761034">(May 18 2018 at 17:44)</a>:</h4>
<p><span class="emoji emoji-1f438" title="frog">:frog:</span> compiles in 12s here vs <span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span> 2.5mins</p>



<a name="126761038"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761038" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761038">(May 18 2018 at 17:44)</a>:</h4>
<p>and..there is that...</p>



<a name="126761044"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761044" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761044">(May 18 2018 at 17:44)</a>:</h4>
<p>:3</p>



<a name="126761052"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761052" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761052">(May 18 2018 at 17:45)</a>:</h4>
<p>we should discuss how to transition the repo</p>



<a name="126761066"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761066" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761066">(May 18 2018 at 17:45)</a>:</h4>
<p>a bit tricky with e.g. <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span>'s in-flight changes but I think those are simple enough we can comment them out and re-do</p>



<a name="126761069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761069">(May 18 2018 at 17:45)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> &lt;3 (you comin to rustfest paris next week BTW ?)</p>



<a name="126761076"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761076" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761076">(May 18 2018 at 17:46)</a>:</h4>
<p>Negative <span class="user-mention" data-user-id="116113">@lqd</span>. That sounds like it might have been a good plan, but didn't think that far ahead.</p>



<a name="126761122"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761122" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761122">(May 18 2018 at 17:46)</a>:</h4>
<p>I'll have a proxy there, though. <span class="user-mention" data-user-id="116852">@Andrea Lattuada</span> should be showing up.</p>



<a name="126761153"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761153" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761153">(May 18 2018 at 17:47)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> we can let Santiago land his work while I convert the pre-pass, then convert the difference, and then you can decide ?</p>



<a name="126761166"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761166" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761166">(May 18 2018 at 17:47)</a>:</h4>
<p>sounds fine</p>



<a name="126761171"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761171" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761171">(May 18 2018 at 17:47)</a>:</h4>
<p>or is that too .. late ?</p>



<a name="126761178"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761178" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761178">(May 18 2018 at 17:47)</a>:</h4>
<p>there's no <em>great</em> hurry— but I would say that if we can reduce the build time from minutes to seconds</p>



<a name="126761179"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761179" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761179">(May 18 2018 at 17:48)</a>:</h4>
<p>it will help push further iteration along :)</p>



<a name="126761218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761218">(May 18 2018 at 17:48)</a>:</h4>
<p>I'd prioritize that personally</p>



<a name="126761230"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761230" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761230">(May 18 2018 at 17:48)</a>:</h4>
<p>anyway I think <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> is busy today</p>



<a name="126761242"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761242" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761242">(May 18 2018 at 17:48)</a>:</h4>
<p>maybe then land these 2 <span class="emoji emoji-1f438" title="frog">:frog:</span> sooner ?</p>



<a name="126761252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761252">(May 18 2018 at 17:49)</a>:</h4>
<p>if you want to prep a PR, I'd merge it</p>



<a name="126761260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761260">(May 18 2018 at 17:49)</a>:</h4>
<p>in any case I think we should consider reframing the 'insensitive' thing not as its own analysis</p>



<a name="126761263"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761263" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761263">(May 18 2018 at 17:49)</a>:</h4>
<p>but as a refinement to timely-opt</p>



<a name="126761266"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761266" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761266">(May 18 2018 at 17:49)</a>:</h4>
<p>ths might be an opportunity to do that reshuffling :)</p>



<a name="126761269"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761269" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761269">(May 18 2018 at 17:49)</a>:</h4>
<p>alright I'll do this between this evening (about to grab dinner) and tomorrow</p>



<a name="126761270"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761270" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761270">(May 18 2018 at 17:49)</a>:</h4>
<p>the only thing I would say is:</p>



<a name="126761275"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761275" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761275">(May 18 2018 at 17:50)</a>:</h4>
<p>do we want to think about macros to make it easier to write?</p>



<a name="126761316"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761316" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761316">(May 18 2018 at 17:50)</a>:</h4>
<p>I guess that can prob wait</p>



<a name="126761331"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761331" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761331">(May 18 2018 at 17:50)</a>:</h4>
<p>particularly as we've already ported it</p>



<a name="126761357"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761357" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761357">(May 18 2018 at 17:51)</a>:</h4>
<p><span class="emoji emoji-1f44d" title="thumbs up">:thumbs_up:</span></p>



<a name="126761367"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761367" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761367">(May 18 2018 at 17:51)</a>:</h4>
<p>(heading off to dinner, bbl)</p>



<a name="126761421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761421">(May 18 2018 at 17:52)</a>:</h4>
<p>The writing was pretty formulaic, btw. But not obviously macro-able. Working through each join and naming each relation I thought I'd need, then seeing what those were (in rustc error messages) and prepping each of them as <code>from_map</code> of their source relation.</p>



<a name="126761430"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761430" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761430">(May 18 2018 at 17:53)</a>:</h4>
<p>yeah I think it's better to wait until we're all done, sort of, and then see if we can add macros or what</p>



<a name="126761443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761443">(May 18 2018 at 17:53)</a>:</h4>
<p>I have a feeling it will evolve further before we're done</p>



<a name="126761445"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761445" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761445">(May 18 2018 at 17:53)</a>:</h4>
<p>no need to bake things in yet</p>



<a name="126761510"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761510" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761510">(May 18 2018 at 17:54)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> IMO yea, macros very last</p>



<a name="126761517"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761517" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> qmx <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761517">(May 18 2018 at 17:54)</a>:</h4>
<p>and I'm really really interested into how this will be done</p>



<a name="126761989"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126761989" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126761989">(May 18 2018 at 18:06)</a>:</h4>
<p>Btw, just added a few more <code>variable_indistinct</code> calls, doing less distinct work, and things actually slowed down a bit (presumably because the non-distinct relations made more work for others). So, might be a bit of exploration to do here.</p>



<a name="126762057"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762057" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762057">(May 18 2018 at 18:08)</a>:</h4>
<p>ah so you were allowing some duplication to slip in basically?</p>



<a name="126762069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762069">(May 18 2018 at 18:08)</a>:</h4>
<p>(I tried to add <code>distinct_total</code> in places I thought might add duplications, but I may have left some because there was a distinct coming later, can't remember)</p>



<a name="126762071"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762071" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762071">(May 18 2018 at 18:08)</a>:</h4>
<p>(and I could have just been wrong)</p>



<a name="126762151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762151">(May 18 2018 at 18:11)</a>:</h4>
<p>I tried removing all distincts except for a few that make all recursive definitions distinct (subset, requires, and dead_can_reach). So, fewer "unnecessary" distincts, but it seems like they may occasionally help performance.</p>



<a name="126762237"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762237" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762237">(May 18 2018 at 18:13)</a>:</h4>
<p>interesting</p>



<a name="126762712"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762712" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762712">(May 18 2018 at 18:24)</a>:</h4>
<p>fwiw, <span class="emoji emoji-1f438" title="frog">:frog:</span> is taking 5,764 rounds to converge, and I think a lot of the distinct work is essentially looking up relatively few facts in larger base relations (vs many facts at once, derived over fewer rounds).</p>



<a name="126762745"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762745" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762745">(May 18 2018 at 18:25)</a>:</h4>
<p>If we had a "large step" cfg relation, even without collapsing down indistinguishable components, it should probably speed things up just due to the way that doing distincts in bulk is more efficient than doing them drawn out over multiple rounds.</p>



<a name="126762754"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762754" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762754">(May 18 2018 at 18:25)</a>:</h4>
<p>what do you mean by "large step"? sort of "reachable"?</p>



<a name="126762757"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762757" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762757">(May 18 2018 at 18:26)</a>:</h4>
<p>like, transitive clsoure? (or some subset of it?)</p>



<a name="126762806"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762806" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762806">(May 18 2018 at 18:26)</a>:</h4>
<p>well, as opposed to "small step" (random PL vocabulary sneaking back in).</p>



<a name="126762815"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762815" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762815">(May 18 2018 at 18:26)</a>:</h4>
<p>oh well <em>that</em> clears everything up</p>



<a name="126762818"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762818" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762818">(May 18 2018 at 18:26)</a>:</h4>
<p>;)</p>



<a name="126762831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762831">(May 18 2018 at 18:27)</a>:</h4>
<p>If we took all the steps that were legit to take at once (i.e. to all points that are not kills, nor at which there is a change in the <code>region_live_at</code> relation).</p>



<a name="126762833"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762833" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762833">(May 18 2018 at 18:27)</a>:</h4>
<p>yeah I have to think about what that means</p>



<a name="126762840"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762840" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762840">(May 18 2018 at 18:27)</a>:</h4>
<p>that is, you can't just compute transitive closure of CFG</p>



<a name="126762847"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762847" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762847">(May 18 2018 at 18:27)</a>:</h4>
<p>but definitely you can imagine taking multiple steps</p>



<a name="126762889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762889">(May 18 2018 at 18:28)</a>:</h4>
<p>on a related note I've wondered if we can aggregate the effects of (e.g.) a basic block</p>



<a name="126762895"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762895" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762895">(May 18 2018 at 18:28)</a>:</h4>
<p>and then compute at the boundaries of basic blocks</p>



<a name="126762906"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762906" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762906">(May 18 2018 at 18:28)</a>:</h4>
<p>(as is often done...)</p>



<a name="126762910"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762910" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762910">(May 18 2018 at 18:28)</a>:</h4>
<p>E.g. we could put together a relation <code>Equiv(P,Q)</code> which is <code>CFG(P,Q)</code> minus any pairs for which <code>RLA(R,P), !RLA(R,Q)</code> or any <code>Killed(R,P)</code> or <code>Killed(R,Q)</code>.</p>



<a name="126762912"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762912" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762912">(May 18 2018 at 18:28)</a>:</h4>
<p>right</p>



<a name="126762959"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762959" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762959">(May 18 2018 at 18:30)</a>:</h4>
<p>that makes sense</p>



<a name="126762985"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126762985" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126762985">(May 18 2018 at 18:30)</a>:</h4>
<p>In principle we could collapse down everything to some representative of <code>Equiv</code>, but even skipping that and just applying all the rules at once (rather than over .. ~100 rounds or whatever) would do something solid.</p>



<a name="126763714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126763714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126763714">(May 18 2018 at 18:47)</a>:</h4>
<p>Fwiw, if you impl the rules</p>
<div class="codehilite"><pre><span></span>        // InEquiv(P,Q) :- CFG(P,Q), Killed(B,P).
        // InEquiv(P,Q) :- CFG(P,Q), RLA(R,P), !RLA(R,Q).
</pre></div>


<p>then there are 6,253 inequiv edges out of 51,896 cfg edges.</p>



<a name="126764331"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764331" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764331">(May 18 2018 at 19:01)</a>:</h4>
<p>we could also process — gross oversimplification — 4 tuples at a time and have the operators use SIMD</p>



<a name="126764375"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764375" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764375">(May 18 2018 at 19:02)</a>:</h4>
<p>(maybe ? sometimes ?)</p>



<a name="126764392"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764392" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764392">(May 18 2018 at 19:02)</a>:</h4>
<p>The EmptyHeaded folks use it to do efficient galloping, but .. I think it involves some aggressive pre-computation for the relations.</p>



<a name="126764411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764411">(May 18 2018 at 19:03)</a>:</h4>
<p>Not exactly multiple tuples at a time, but searching for multiple matches for one tuple at a time.</p>



<a name="126764427"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764427" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764427">(May 18 2018 at 19:04)</a>:</h4>
<p>Lots to do, but I think with those equiv vs cfg numbers, there is a 10x speed-up waiting if one consolidates indistinguishable components of the cfg.</p>



<a name="126764450"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764450" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764450">(May 18 2018 at 19:04)</a>:</h4>
<p>I also remember LegoBase (I think) wrt to merging of operators and the likes</p>



<a name="126764475"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764475" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764475">(May 18 2018 at 19:04)</a>:</h4>
<p>ofc they staged queries but we know them ahead of time</p>



<a name="126764492"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764492" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764492">(May 18 2018 at 19:05)</a>:</h4>
<p>agreed, the cfg looks like a good avenue</p>



<a name="126764563"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764563" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764563">(May 18 2018 at 19:07)</a>:</h4>
<p>maybe could also try automatically different orders for the rules "evaluation"</p>



<a name="126764608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764608">(May 18 2018 at 19:08)</a>:</h4>
<p>(but that may overfitting to clap in particular, who knows)</p>



<a name="126764630"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764630" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764630">(May 18 2018 at 19:09)</a>:</h4>
<p>as you said reducing the millions of tuples flying about will surely always be the best bet :)</p>



<a name="126764893"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764893" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764893">(May 18 2018 at 19:15)</a>:</h4>
<p>fwiw, breakdown of total tuples produced (in the 9.5s or so):</p>
<div class="codehilite"><pre><span></span>FINAL: &quot;region_live_at_p&quot;   1076158
FINAL: &quot;inequiv&quot;    1154468
FINAL: &quot;killed_p&quot;   980
FINAL: &quot;cfg_edge_p&quot; 51896
FINAL: &quot;region_live_at&quot; 1076158
FINAL: &quot;dead_can_reach_live_r1pq&quot;   38684
FINAL: &quot;dead_can_reach_live&quot;    38684
FINAL: &quot;dead_can_reach_r2q&quot; 194748
FINAL: &quot;dead_can_reach_1&quot;   156064
FINAL: &quot;dead_can_reach&quot; 194748
FINAL: &quot;dead_can_reach_origins&quot; 43222
FINAL: &quot;dead_region_requires_rpq&quot;   50555
FINAL: &quot;dead_region_requires_2&quot; 907164
FINAL: &quot;dead_region_requires_1&quot; 858650
FINAL: &quot;dead_region_requires&quot;   50555
FINAL: &quot;live_to_dead_regions_r2pq&quot;  38098
FINAL: &quot;live_to_dead_regions_p&quot; 38098
FINAL: &quot;live_to_dead_regions_2&quot; 2719027
FINAL: &quot;live_to_dead_regions_1&quot; 2931089
FINAL: &quot;live_to_dead_regions&quot;   38098
FINAL: &quot;requires_rp&quot;    858674
FINAL: &quot;requires_bp&quot;    858674
FINAL: &quot;requires_2&quot; 859328
FINAL: &quot;requires_1&quot; 858650
FINAL: &quot;requires&quot;   858674
FINAL: &quot;subset_p&quot;   2714121
FINAL: &quot;subset_r2p&quot; 2714121
FINAL: &quot;subset_r1p&quot; 2714121
FINAL: &quot;subset_2&quot;   2661662
FINAL: &quot;subset_1&quot;   2785151
FINAL: &quot;subset&quot; 2714121
FINAL: &quot;subset&quot; 2714121
FINAL: &quot;subset_1&quot;   2785151
FINAL: &quot;subset_2&quot;   2661662
FINAL: &quot;subset_r1p&quot; 2714121
FINAL: &quot;subset_r2p&quot; 2714121
FINAL: &quot;subset_p&quot;   2714121
FINAL: &quot;requires&quot;   858674
FINAL: &quot;requires_1&quot; 858650
FINAL: &quot;requires_2&quot; 859328
FINAL: &quot;requires_bp&quot;    858674
FINAL: &quot;requires_rp&quot;    858674
FINAL: &quot;borrow_live_at&quot; 0
FINAL: &quot;live_to_dead_regions&quot;   38098
FINAL: &quot;live_to_dead_regions_1&quot; 2931089
FINAL: &quot;live_to_dead_regions_2&quot; 2719027
FINAL: &quot;live_to_dead_regions_p&quot; 38098
FINAL: &quot;live_to_dead_regions_r2pq&quot;  38098
FINAL: &quot;dead_region_requires&quot;   50555
FINAL: &quot;dead_region_requires_1&quot; 858650
FINAL: &quot;dead_region_requires_2&quot; 907164
FINAL: &quot;dead_region_requires_rpq&quot;   50555
FINAL: &quot;dead_can_reach_origins&quot; 43222
FINAL: &quot;dead_can_reach&quot; 194748
FINAL: &quot;dead_can_reach_1&quot;   156064
FINAL: &quot;dead_can_reach_r2q&quot; 194748
FINAL: &quot;dead_can_reach_live&quot;    38684
FINAL: &quot;dead_can_reach_live_r1pq&quot;   38684
FINAL: &quot;region_live_at&quot; 1076158
FINAL: &quot;cfg_edge_p&quot; 51896
</pre></div>


<p>NB: each relation seems to be dropped twice. Not sure what is up with that.</p>



<a name="126764946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126764946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126764946">(May 18 2018 at 19:17)</a>:</h4>
<p>Oh, the boxed versions getting dropped (the ones <code>Iteration</code> holds on to).</p>



<a name="126765101"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765101" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765101">(May 18 2018 at 19:20)</a>:</h4>
<p>Ooo, optimization:</p>
<div class="codehilite"><pre><span></span>        // live_to_dead_regions(R1, R2, P, Q) :-
        //   subset(R1, R2, P),
        //   cfg_edge(P, Q),
        //   region_live_at(R1, Q),
        //   !region_live_at(R2, Q).
</pre></div>


<p>should be </p>
<div class="codehilite"><pre><span></span>        // live_to_dead_regions(R1, R2, P, Q) :-
        //   subset(R1, R2, P),
        //   cfg_edge(P, Q),
        //   !region_live_at(R2, Q).
        //   region_live_at(R1, Q),
</pre></div>



<a name="126765116"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765116" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765116">(May 18 2018 at 19:20)</a>:</h4>
<p>should we try and get all antijoins as soon as possible ?</p>



<a name="126765131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765131">(May 18 2018 at 19:21)</a>:</h4>
<p>the earliest we filter the better</p>



<a name="126765133"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765133" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765133">(May 18 2018 at 19:21)</a>:</h4>
<p>Not sure, but looking at the sizes up above, there is a 2.9M -&gt; 2.7M change with the semijoin, and then a 2.7M -&gt; 38K change with the antijoin. Not sure if it is general, though.</p>



<a name="126765177"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765177" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765177">(May 18 2018 at 19:22)</a>:</h4>
<p>Both the antijoin and the semijoin filter, so it isn't as obvious as that unfortunately. (I made the mistake of thinking it was when I posted "Ooo, optimization").</p>



<a name="126765183"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765183" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Andrea Lattuada <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765183">(May 18 2018 at 19:22)</a>:</h4>
<p><span class="emoji emoji-1f44b" title="wave">:wave:</span> Hi folks! Frank’s proxy here :) looking forward to RustFest, and happy to chat about Differential Dataflow-related things!</p>



<a name="126765189"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765189" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765189">(May 18 2018 at 19:22)</a>:</h4>
<p>true, so did I</p>



<a name="126765202"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765202" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765202">(May 18 2018 at 19:23)</a>:</h4>
<p>But in this case I think that joining subset and cfg first might be wrong (should probably join cfg and rla, then semijoin against subset).</p>



<a name="126765205"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765205" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765205">(May 18 2018 at 19:24)</a>:</h4>
<p>Heyo <span class="user-mention" data-user-id="116852">@Andrea Lattuada</span> !</p>



<a name="126765750"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765750" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765750">(May 18 2018 at 19:31)</a>:</h4>
<p>Pondering whether I should just go to Paris anyhow and see what is going on...</p>



<a name="126765795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765795">(May 18 2018 at 19:32)</a>:</h4>
<p>tickets were sold out, but I saw them talking about last minute tickets</p>



<a name="126765868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765868">(May 18 2018 at 19:34)</a>:</h4>
<p>(I think Felix is also part of the organization <span class="emoji emoji-1f914" title="thinking face">:thinking_face:</span>)</p>



<a name="126765879"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765879" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765879">(May 18 2018 at 19:34)</a>:</h4>
<p>Was mostly just thinking about going to Paris. &gt;.&gt;</p>



<a name="126765897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126765897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126765897">(May 18 2018 at 19:35)</a>:</h4>
<p>lol</p>



<a name="126766042"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766042" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766042">(May 18 2018 at 19:38)</a>:</h4>
<blockquote>
<p>Was mostly just thinking about going to Paris. &gt;.&gt;</p>
</blockquote>
<p>I agree with this reasoning</p>



<a name="126766045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766045">(May 18 2018 at 19:38)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> do you want me to replace <a href="http://naive.rs" target="_blank" title="http://naive.rs">naive.rs</a> with the <span class="emoji emoji-1f438" title="frog">:frog:</span> naive, and in general remove DD usage ? what to do about location_insensitive, want me to translate with the PR ?</p>



<a name="126766053"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766053" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766053">(May 18 2018 at 19:39)</a>:</h4>
<p>or later</p>



<a name="126766056"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766056" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766056">(May 18 2018 at 19:39)</a>:</h4>
<blockquote>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> do you want me to replace <a href="http://naive.rs" target="_blank" title="http://naive.rs">naive.rs</a> with the <span class="emoji emoji-1f438" title="frog">:frog:</span> naive, and in general remove DD usage ? </p>
</blockquote>
<p>yes</p>



<a name="126766065"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766065" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766065">(May 18 2018 at 19:39)</a>:</h4>
<p>re: location-insensitve, i'd be ok with commenting it out for now</p>



<a name="126766067"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766067" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766067">(May 18 2018 at 19:39)</a>:</h4>
<p>translating it would be a good exercise</p>



<a name="126766069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766069">(May 18 2018 at 19:39)</a>:</h4>
<p>for someone else to get familiar :)</p>



<a name="126766149"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766149" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766149">(May 18 2018 at 19:41)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> do you want me to git commit with your GH email as the author ? since you are, you know, the author</p>



<a name="126766227"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766227" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766227">(May 18 2018 at 19:42)</a>:</h4>
<p>Oh, however you like <span class="user-mention" data-user-id="116113">@lqd</span>. I'm happy to donate authorship of that stuff. Whichever way is least likely to cause problems down the road (if the commits are serious about authorship and don't like screwing around, then probably yes).</p>



<a name="126766242"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766242" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766242">(May 18 2018 at 19:43)</a>:</h4>
<p>I wouldn't know myself, maybe <span class="user-mention" data-user-id="116009">@nikomatsakis</span> can chime in ?</p>



<a name="126766247"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766247" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766247">(May 18 2018 at 19:43)</a>:</h4>
<p>I don't think it matters either way</p>



<a name="126766297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766297">(May 18 2018 at 19:44)</a>:</h4>
<p>in general we state that, if you open a PR, you are claiming willingness to release that code under the license</p>



<a name="126766309"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766309" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766309">(May 18 2018 at 19:44)</a>:</h4>
<p>maybe I just integrate <span class="emoji emoji-1f438" title="frog">:frog:</span> with our naive version and then frank opens a PR for the opt version ?</p>



<a name="126766319"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766319" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766319">(May 18 2018 at 19:45)</a>:</h4>
<p>either wfm</p>



<a name="126766322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766322">(May 18 2018 at 19:45)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> may not want to :)</p>



<a name="126766324"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766324" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766324">(May 18 2018 at 19:45)</a>:</h4>
<p>:)</p>



<a name="126766327"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766327" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766327">(May 18 2018 at 19:45)</a>:</h4>
<p>oh geez, yeah that means I have to figure out how to PR at you all without deleting everything else in your repo.</p>



<a name="126766330"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766330" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766330">(May 18 2018 at 19:45)</a>:</h4>
<p>heh that was sort of the reaction I expected</p>



<a name="126766332"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766332" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766332">(May 18 2018 at 19:45)</a>:</h4>
<p>I say <span class="user-mention" data-user-id="116113">@lqd</span> just do it all and credit <span class="user-mention" data-user-id="116609">@Frank McSherry</span> in the comment</p>



<a name="126766333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126766333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126766333">(May 18 2018 at 19:46)</a>:</h4>
<p>"hrm, looks like someone reverted all of those PartialOrd and Ord changes from several versions back..."</p>



<a name="126767021"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767021" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767021">(May 18 2018 at 20:01)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> is the gist from a couple hours back the version I should add ?</p>



<a name="126767174"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767174" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767174">(May 18 2018 at 20:05)</a>:</h4>
<p>probably, lemme check. the one just post-<code>variable_indistinct</code> is good.</p>



<a name="126767236"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767236" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767236">(May 18 2018 at 20:06)</a>:</h4>
<p>yeah <a href="https://gist.github.com/frankmcsherry/adb9ed3c433eb9f4ddaf591ad9888dea" target="_blank" title="https://gist.github.com/frankmcsherry/adb9ed3c433eb9f4ddaf591ad9888dea">https://gist.github.com/frankmcsherry/adb9ed3c433eb9f4ddaf591ad9888dea</a> is right with the indistincts</p>



<a name="126767245"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767245" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767245">(May 18 2018 at 20:07)</a>:</h4>
<p>seems right to me.</p>



<a name="126767257"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767257" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767257">(May 18 2018 at 20:07)</a>:</h4>
<p>I tidied it a bit (no need to have a scope to get <code>bla</code>) but it's essentially the right thing for now.</p>



<a name="126767261"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767261" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767261">(May 18 2018 at 20:07)</a>:</h4>
<p>I suspect it will continue to evolve a bit.</p>



<a name="126767263"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767263" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767263">(May 18 2018 at 20:07)</a>:</h4>
<p>so without the antijoins ordering, or removing/adding some distincts you were mentioning</p>



<a name="126767314"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767314" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767314">(May 18 2018 at 20:08)</a>:</h4>
<p>yeah and niko really wants to switch away from <code>bla</code>s ;)</p>



<a name="126767316"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767316" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767316">(May 18 2018 at 20:08)</a>:</h4>
<p>no, haven't tried out the antijoin order change, and the other distincts were small order changes (0.1s-ish).</p>



<a name="126767322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767322">(May 18 2018 at 20:08)</a>:</h4>
<p>alright, thank you</p>



<a name="126767332"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767332" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767332">(May 18 2018 at 20:09)</a>:</h4>
<p>No worries, thanks for guiding all the work around!</p>



<a name="126767404"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767404" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767404">(May 18 2018 at 20:10)</a>:</h4>
<p>you did all the work! :p</p>



<a name="126767889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126767889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126767889">(May 18 2018 at 20:23)</a>:</h4>
<p>I'm loving the 20x reduction in compile time :3</p>



<a name="126768835"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126768835" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126768835">(May 18 2018 at 20:46)</a>:</h4>
<p>/me fixes his "Datafro" typos</p>



<a name="126769603"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769603" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769603">(May 18 2018 at 21:06)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> unless I made some git mistakes, it might be time to <span class="emoji emoji-1f438" title="frog">:frog:</span> <a href="https://github.com/rust-lang-nursery/polonius/pull/36" target="_blank" title="https://github.com/rust-lang-nursery/polonius/pull/36">https://github.com/rust-lang-nursery/polonius/pull/36</a></p>



<a name="126769617"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769617" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769617">(May 18 2018 at 21:07)</a>:</h4>
<p>love the PR title :)</p>



<a name="126769620"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769620" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769620">(May 18 2018 at 21:07)</a>:</h4>
<p>:)</p>



<a name="126769624"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769624" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769624">(May 18 2018 at 21:07)</a>:</h4>
<p><span class="emoji emoji-1f438" title="frog">:frog:</span></p>



<a name="126769663"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769663" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769663">(May 18 2018 at 21:08)</a>:</h4>
<p>Kinda thinking Niko gets 50% authorship too, since I copy/pasted all of his comments.</p>



<a name="126769680"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769680" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769680">(May 18 2018 at 21:08)</a>:</h4>
<p>I did 50% of those copy/pastes frank</p>



<a name="126769694"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769694" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769694">(May 18 2018 at 21:09)</a>:</h4>
<p>I think I'll double-check by hand it produces the same tuples, then merge</p>



<a name="126769703"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769703" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769703">(May 18 2018 at 21:09)</a>:</h4>
<p>though I should really read the core carefully at some point</p>



<a name="126769718"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769718" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769718">(May 18 2018 at 21:09)</a>:</h4>
<p>(kidding ofc, but I actually did integrate more of the timelyopt comments so we don't lose them)</p>



<a name="126769725"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769725" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769725">(May 18 2018 at 21:09)</a>:</h4>
<p>def take a peek; I unsafe cast a few u32s to usizes, but pretty sure it's all good</p>



<a name="126769786"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769786" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769786">(May 18 2018 at 21:10)</a>:</h4>
<p>(( i don't do anything like that, but since the datafrog crate is under my control, .. muaaahaaha. ))</p>



<a name="126769788"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769788" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769788">(May 18 2018 at 21:10)</a>:</h4>
<p>agreed wrt to manual checking first</p>



<a name="126769790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769790">(May 18 2018 at 21:10)</a>:</h4>
<p>yeah, that'll just have to wait till later in the day</p>



<a name="126769795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769795">(May 18 2018 at 21:10)</a>:</h4>
<p>anyway nice job y'all!</p>



<a name="126769852"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769852" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769852">(May 18 2018 at 21:12)</a>:</h4>
<p>it will be nice to have <span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span> vs <span class="emoji emoji-1f438" title="frog">:frog:</span> numbers from your machine as well</p>



<a name="126769876"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769876" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769876">(May 18 2018 at 21:13)</a>:</h4>
<p>ping <span class="user-mention" data-user-id="116083">@pnkfelix</span> — this branch may change the integration story</p>



<a name="126769930"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769930" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769930">(May 18 2018 at 21:14)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> doing a local build now</p>



<a name="126769933"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769933" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769933">(May 18 2018 at 21:14)</a>:</h4>
<p>I also used the word bespoke as I know you like it</p>



<a name="126769937"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769937" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769937">(May 18 2018 at 21:14)</a>:</h4>
<p>lol</p>



<a name="126769940"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769940" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769940">(May 18 2018 at 21:14)</a>:</h4>
<p>I actually picked it up from <span class="user-mention" data-user-id="116609">@Frank McSherry</span> ;)</p>



<a name="126769943"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769943" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769943">(May 18 2018 at 21:14)</a>:</h4>
<p>I think I recall him talking about "bespoke union find analyses" or something</p>



<a name="126769946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769946">(May 18 2018 at 21:14)</a>:</h4>
<p>it all makes sense</p>



<a name="126769954"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769954" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769954">(May 18 2018 at 21:15)</a>:</h4>
<p><span class="emoji emoji-2b55" title="heavy large circle">:heavy_large_circle:</span></p>



<a name="126769964"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126769964" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126769964">(May 18 2018 at 21:15)</a>:</h4>
<p>(seriously tho, CHR might be good for bespoke union find (ofc Chalk comes to mind))</p>



<a name="126770035"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126770035" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126770035">(May 18 2018 at 21:17)</a>:</h4>
<p>(I'll redirect most talks about this to eternaleye, who would precisely explain why, better than I could — this goes for this topic and many others ;)</p>



<a name="126770087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126770087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126770087">(May 18 2018 at 21:18)</a>:</h4>
<p>"CHR, what else"</p>



<a name="126770836"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126770836" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126770836">(May 18 2018 at 21:40)</a>:</h4>
<p>if all goes well, after merging we should also create an issue for the location-insensitive <span class="emoji emoji-1f438" title="frog">:frog:</span> conversion, I left some comments about that in the PR as well</p>



<a name="126774464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126774464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126774464">(May 18 2018 at 23:18)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> pushed another cleanup commit — measurements are variable on my laptop, but it's a 5% max improvement</p>



<a name="126774598"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126774598" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126774598">(May 18 2018 at 23:22)</a>:</h4>
<p>I also suspect mw, zoxc, or njn will be able to improve all this :)</p>



<a name="126775096"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126775096" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126775096">(May 18 2018 at 23:39)</a>:</h4>
<p>had a chance to test it locally:</p>
<table>
<thead>
<tr>
<th>Version</th>
<th>Compile Time</th>
<th>Run Time</th>
</tr>
</thead>
<tbody>
<tr>
<td><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span></td>
<td>210s</td>
<td>14.191s</td>
</tr>
<tr>
<td><span class="emoji emoji-1f438" title="frog">:frog:</span></td>
<td>11s</td>
<td>8s</td>
</tr>
</tbody>
</table>



<a name="126775097"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126775097" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126775097">(May 18 2018 at 23:40)</a>:</h4>
<p>I'd...say we have a winner :)</p>



<a name="126775140"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126775140" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126775140">(May 18 2018 at 23:40)</a>:</h4>
<p>/me afk</p>



<a name="126775310"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126775310" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126775310">(May 18 2018 at 23:47)</a>:</h4>
<p>we rarely saw results with more than one <span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span> worker AFAICT ?</p>



<a name="126776259"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126776259" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126776259">(May 19 2018 at 00:19)</a>:</h4>
<p>With threads (8 core machine):</p>
<table>
<thead>
<tr>
<th>Version</th>
<th>Workers</th>
<th>Compile Time</th>
<th>Run Time</th>
</tr>
</thead>
<tbody>
<tr>
<td><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span></td>
<td>1</td>
<td>210s</td>
<td>14.191s</td>
</tr>
<tr>
<td><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span></td>
<td>2</td>
<td>210s</td>
<td>11.035s</td>
</tr>
<tr>
<td><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span></td>
<td>4</td>
<td>210s</td>
<td>8.898s</td>
</tr>
<tr>
<td><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span></td>
<td>8</td>
<td>210s</td>
<td>7.953s</td>
</tr>
<tr>
<td><span class="emoji emoji-1f438" title="frog">:frog:</span></td>
<td>1</td>
<td>11s</td>
<td>8s</td>
</tr>
</tbody>
</table>



<a name="126776378"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126776378" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126776378">(May 19 2018 at 00:23)</a>:</h4>
<p>oh awesome thank you :)</p>



<a name="126787211"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126787211" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126787211">(May 19 2018 at 07:34)</a>:</h4>
<p>Comment on the PR, <span class="user-mention" data-user-id="116113">@lqd</span>. Just noticed another 1.5s we can shave off by further <code>_indistinct()</code> use (for some reason I forgot to do it to all of the <code>requires_</code> variants).</p>



<a name="126788061"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126788061" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126788061">(May 19 2018 at 08:11)</a>:</h4>
<p>oh very nice <span class="emoji emoji-1f642" title="simple smile">:simple_smile:</span></p>



<a name="126788159"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126788159" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126788159">(May 19 2018 at 08:15)</a>:</h4>
<p>I also tried "compressed", with the equiv / inequiv rules. As you mentioned, it can likely be a big win to use that successfully</p>



<a name="126788916"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126788916" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126788916">(May 19 2018 at 08:45)</a>:</h4>
<p>it is on everyone's mind <span class="emoji emoji-1f603" title="smiley">:smiley:</span> <a href="https://twitter.com/martinkl/status/997500551746158593" target="_blank" title="https://twitter.com/martinkl/status/997500551746158593">https://twitter.com/martinkl/status/997500551746158593</a></p>
<div class="inline-preview-twitter"><div class="twitter-tweet"><a href="https://twitter.com/martinkl/status/997500551746158593" target="_blank"><img class="twitter-avatar" src="https://pbs.twimg.com/profile_images/683377237547192321/ZFAx7ted_normal.jpg"></a><p>I am looking for Datalog engines that can perform incremental view maintenance. I know of <a href="https://twitter.com/logicblox" target="_blank" title="https://twitter.com/logicblox">@logicblox</a>; any others? (cc <a href="https://twitter.com/palvaro" target="_blank" title="https://twitter.com/palvaro">@palvaro</a>)</p><span>- Martin Kleppmann (@martinkl)</span></div></div>



<a name="126788972"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126788972" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126788972">(May 19 2018 at 08:47)</a>:</h4>
<p>which reminds me we could also look at Eve's Rust runtime, since they supposedly did just that</p>



<a name="126789092"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126789092" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126789092">(May 19 2018 at 08:51)</a>:</h4>
<p>(Expressing CRDTs using Datalog <span class="emoji emoji-1f914" title="thinking face">:thinking_face:</span>)</p>



<a name="126789487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126789487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126789487">(May 19 2018 at 09:07)</a>:</h4>
<p>I also pushed the requires_* indistincts, 10-15% gain for free</p>



<a name="126789583"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126789583" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126789583">(May 19 2018 at 09:11)</a>:</h4>
<blockquote>
<p>Comment on the PR, <span class="user-mention" data-user-id="116113">@lqd</span>. Just noticed another 1.5s we can shave off by further <code>_indistinct()</code> use (for some reason I forgot to do it to all of the <code>requires_</code> variants).</p>
</blockquote>
<p>taking that into account, I get:</p>
<p>With threads (8 core machine):</p>
<table>
<thead>
<tr>
<th>Version</th>
<th>Workers</th>
<th>Compile Time</th>
<th>Run Time</th>
</tr>
</thead>
<tbody>
<tr>
<td><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span></td>
<td>1</td>
<td>210s</td>
<td>14.191s</td>
</tr>
<tr>
<td><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span></td>
<td>2</td>
<td>210s</td>
<td>11.035s</td>
</tr>
<tr>
<td><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span></td>
<td>4</td>
<td>210s</td>
<td>8.898s</td>
</tr>
<tr>
<td><span class="emoji emoji-23f2" title="timer clock">:timer_clock:</span></td>
<td>8</td>
<td>210s</td>
<td>7.953s</td>
</tr>
<tr>
<td><span class="emoji emoji-1f438" title="frog">:frog:</span></td>
<td>1</td>
<td>11s</td>
<td>7.154s</td>
</tr>
</tbody>
</table>



<a name="126790073"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790073" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790073">(May 19 2018 at 09:30)</a>:</h4>
<p>nice :)</p>



<a name="126790121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790121">(May 19 2018 at 09:32)</a>:</h4>
<p>I'm making a few tweaks as I go; just got another 5% win incidentally =)</p>



<a name="126790127"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790127" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790127">(May 19 2018 at 09:33)</a>:</h4>
<p>if we could "uncompress/unpack" from the InEquiv points, it would be <span class="emoji emoji-1f44d" title="thumbs up">:thumbs_up:</span> — unless I'm horribly mistaken, computing these points is very fast, using these edges instead of the full cfg is very fast, but don't know yet how to get to the original results back (but that would make it &lt; 2s on clap)</p>



<a name="126790128"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790128" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790128">(May 19 2018 at 09:33)</a>:</h4>
<p>(so down to 6.8s on my machine)</p>



<a name="126790166"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790166" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790166">(May 19 2018 at 09:34)</a>:</h4>
<p>I've not yet gotten to that point, still going over the relations bit by bit, but yes that was my intuition when filing #20</p>



<a name="126790168"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790168" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790168">(May 19 2018 at 09:34)</a>:</h4>
<p>("compressing" is &lt;1s on clap on my slow laptop)</p>



<a name="126790221"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790221" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790221">(May 19 2018 at 09:37)</a>:</h4>
<p>hacky, and full of clones, but here's the InEquiv computation <a href="https://gist.github.com/lqd/743abae3913bc3785a955b749289ca03" target="_blank" title="https://gist.github.com/lqd/743abae3913bc3785a955b749289ca03">https://gist.github.com/lqd/743abae3913bc3785a955b749289ca03</a></p>



<a name="126790260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790260">(May 19 2018 at 09:38)</a>:</h4>
<p>lovin' the faster compile times</p>



<a name="126790268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790268">(May 19 2018 at 09:39)</a>:</h4>
<p>oh yeah your 5% win, a bit like adapting timelyopt's previous "micro-optimization arrangements" <span class="emoji emoji-1f44d" title="thumbs up">:thumbs_up:</span> there were a couple of those before</p>



<a name="126790375"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790375" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790375">(May 19 2018 at 09:43)</a>:</h4>
<p>yeah, although I tried in one other cases and it was not a win</p>



<a name="126790377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790377">(May 19 2018 at 09:43)</a>:</h4>
<p>I'm not really sure why</p>



<a name="126790411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790411">(May 19 2018 at 09:44)</a>:</h4>
<p>that is, removing <code>dead_can_reach_r2q</code> went from 6.8s to 6.9s</p>



<a name="126790419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790419">(May 19 2018 at 09:44)</a>:</h4>
<p>(almost sounds like noise, but it was quite consistent)</p>



<a name="126790421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790421">(May 19 2018 at 09:44)</a>:</h4>
<p>at this point though I would say</p>



<a name="126790423"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790423" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790423">(May 19 2018 at 09:44)</a>:</h4>
<p>we need more tests</p>



<a name="126790425"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790425" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790425">(May 19 2018 at 09:44)</a>:</h4>
<p>that is, our "benchmark suite" is rather limited :)</p>



<a name="126790427"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790427" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790427">(May 19 2018 at 09:44)</a>:</h4>
<p>anyway, I read through the datafrog-opt code now fairly closely</p>



<a name="126790428"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790428" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790428">(May 19 2018 at 09:44)</a>:</h4>
<p>looks great!</p>



<a name="126790474"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790474" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790474">(May 19 2018 at 09:46)</a>:</h4>
<p>just gonna run rustfmt then merge</p>



<a name="126790478"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790478" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790478">(May 19 2018 at 09:46)</a>:</h4>
<p>yay</p>



<a name="126790481"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790481" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790481">(May 19 2018 at 09:46)</a>:</h4>
<p>insert appropriate celebratory emojis here</p>



<a name="126790487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790487">(May 19 2018 at 09:47)</a>:</h4>
<p><span class="emoji emoji-1f370" title="cake">:cake:</span></p>



<a name="126790489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790489">(May 19 2018 at 09:47)</a>:</h4>
<p><span class="emoji emoji-1f389" title="tada">:tada:</span> <span class="emoji emoji-1f44f" title="clap">:clap:</span> <span class="emoji emoji-1f91d" title="handshake">:handshake:</span> <span class="emoji emoji-1f44d" title="thumbs up">:thumbs_up:</span> <span class="emoji emoji-1f38a" title="confetti ball">:confetti_ball:</span></p>



<a name="126790490"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790490" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790490">(May 19 2018 at 09:47)</a>:</h4>
<p>and of course <span class="emoji emoji-1f438" title="frog">:frog:</span></p>



<a name="126790535"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790535" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790535">(May 19 2018 at 09:49)</a>:</h4>
<p>ha yes</p>



<a name="126790536"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790536" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790536">(May 19 2018 at 09:49)</a>:</h4>
<p>it'd be nice to move from a git dependency to a <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a> dep</p>



<a name="126790537"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790537" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790537">(May 19 2018 at 09:49)</a>:</h4>
<p>yup naive has 3 iterations but doesn't really need it</p>



<a name="126790538"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790538" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790538">(May 19 2018 at 09:49)</a>:</h4>
<p>but I don't know if the latest datafrog is fully published there?</p>



<a name="126790539"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790539" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790539">(May 19 2018 at 09:49)</a>:</h4>
<p>anyway not a big thing</p>



<a name="126790596"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790596" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790596">(May 19 2018 at 09:50)</a>:</h4>
<p>and because of that the indices were not shared like they are in opt (plus I didn't know I could collapse them into one at the time, but we can do that if you want)</p>



<a name="126790600"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790600" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790600">(May 19 2018 at 09:50)</a>:</h4>
<p>I don't think frank has published it on <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a></p>



<a name="126790652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790652">(May 19 2018 at 09:53)</a>:</h4>
<blockquote>
<p>and because of that the indices were not shared like they are in opt (plus I didn't know I could collapse them into one at the time, but we can do that if you want)</p>
</blockquote>
<p>I'd like to keep naive as "obviously correct" as we can, but actually one iteration is somehow more "obvious" than multiple, so perhaps it's worth it.</p>



<a name="126790654"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790654" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790654">(May 19 2018 at 09:53)</a>:</h4>
<p>that is, in "the datalog source", we don't have multiple iterations</p>



<a name="126790696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790696">(May 19 2018 at 09:54)</a>:</h4>
<p>true</p>



<a name="126790697"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790697" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790697">(May 19 2018 at 09:54)</a>:</h4>
<p>and it'd probably be more idiomatic to actually share those indices</p>



<a name="126790698"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790698" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790698">(May 19 2018 at 09:54)</a>:</h4>
<p>more datafroggy</p>



<a name="126790709"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790709" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790709">(May 19 2018 at 09:55)</a>:</h4>
<p>I can take care of it whenever, but it would also be a "good first issue" for others, get into the API, with an executable test with known and comparable results</p>



<a name="126790748"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790748" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790748">(May 19 2018 at 09:57)</a>:</h4>
<p>true — can you maybe file an issue around it?</p>



<a name="126790753"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790753" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790753">(May 19 2018 at 09:57)</a>:</h4>
<p>will do</p>



<a name="126790755"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790755" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790755">(May 19 2018 at 09:57)</a>:</h4>
<p>and there also needs a similar one for location-insensitive work</p>



<a name="126790794"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790794" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790794">(May 19 2018 at 09:58)</a>:</h4>
<p>yep</p>



<a name="126790847"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790847" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790847">(May 19 2018 at 10:00)</a>:</h4>
<p>I wanted to write something like that last night in any case — I want to think about whether the location-insensitive work ought to be done as a 'separate analysis' or as part of the datafrog-opt analysis. (maybe we should just clone datafrog-opt and layer the location-insensitive in there, but that might not make sense either.)</p>



<a name="126790853"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790853" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790853">(May 19 2018 at 10:01)</a>:</h4>
<p>I probably won't get much more time to poke at this during the weekend, but we can pick it up on monday I suppose</p>



<a name="126790856"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790856" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790856">(May 19 2018 at 10:01)</a>:</h4>
<p>I guess to start it makes sense to port the location-insensitive analysis as it was</p>



<a name="126790944"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126790944" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126790944">(May 19 2018 at 10:05)</a>:</h4>
<p>maybe others will get to it before then, otherwise yeah I can do both this weekend, or get back to it after that</p>



<a name="126791034"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791034" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791034">(May 19 2018 at 10:08)</a>:</h4>
<p>either way; it seems fine to me to let e.g. <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> or others do some of it, to get a feel for how <span class="emoji emoji-1f438" title="frog">:frog:</span> works</p>



<a name="126791126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791126">(May 19 2018 at 10:12)</a>:</h4>
<p>I can't add labels but here's the first issue <a href="https://github.com/rust-lang-nursery/polonius/issues/37" target="_blank" title="https://github.com/rust-lang-nursery/polonius/issues/37">https://github.com/rust-lang-nursery/polonius/issues/37</a></p>



<a name="126791131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791131">(May 19 2018 at 10:13)</a>:</h4>
<p>do we need a new issue for the location-insensitive port or would I mention it on <a href="https://github.com/rust-lang-nursery/polonius/issues/29" target="_blank" title="https://github.com/rust-lang-nursery/polonius/issues/29">https://github.com/rust-lang-nursery/polonius/issues/29</a> ?</p>



<a name="126791170"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791170" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791170">(May 19 2018 at 10:14)</a>:</h4>
<p>I'd say just mention it in the existing issue</p>



<a name="126791379"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791379" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791379">(May 19 2018 at 10:23)</a>:</h4>
<p>alright done, <a href="https://github.com/rust-lang-nursery/polonius/issues/29#issuecomment-390395464" target="_blank" title="https://github.com/rust-lang-nursery/polonius/issues/29#issuecomment-390395464">https://github.com/rust-lang-nursery/polonius/issues/29#issuecomment-390395464</a></p>



<a name="126791418"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791418" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791418">(May 19 2018 at 10:24)</a>:</h4>
<p>I'm up for pushing at <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a> whenever. I just pushed a version with a perf improvement. Nothing for borrow_check, but another example (<a href="http://graspan1.rs" target="_blank" title="http://graspan1.rs">graspan1.rs</a>) got about 2x out of it. It .. lead me to something I'm not sure I know the right idiom for:</p>
<p>The <code>Relation</code> type has a <code>From</code> implementation for <code>I: IntoIterator</code>, which is pretty general but not actually the right implementation for <code>Vec&lt;Tuple&gt;</code>, because it drains the iterator and collects it into a new allocation. It seems like you'd also want a <code>From</code> for <code>Vec&lt;Tuple&gt;</code>, perhaps using specialization (?). For now, there is a <code>from_vec()</code> method, and using that gives some wins for me in the other project (much larger inputs relative to derived facts).</p>



<a name="126791427"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791427" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791427">(May 19 2018 at 10:25)</a>:</h4>
<p>I've also prepped a blog post about the engine, meant to explain things from the ground up. Much editing to do, but here is a peek: <a href="https://github.com/frankmcsherry/blog/blob/master/posts/2018-05-19.md" target="_blank" title="https://github.com/frankmcsherry/blog/blob/master/posts/2018-05-19.md">https://github.com/frankmcsherry/blog/blob/master/posts/2018-05-19.md</a></p>



<a name="126791428"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791428" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791428">(May 19 2018 at 10:25)</a>:</h4>
<p>I was seeing a number of <code>merges</code>with empty relations, removing those out didn't make much a difference I think, but could be wrong</p>



<a name="126791443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791443">(May 19 2018 at 10:26)</a>:</h4>
<p>If any of <span class="user-mention" data-user-id="116113">@lqd</span> , <span class="user-mention" data-user-id="116009">@nikomatsakis</span> , <span class="user-mention" data-user-id="116013">@qmx</span> do not want to be ack'd in it, please let me know and I will not.</p>



<a name="126791469"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791469" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791469">(May 19 2018 at 10:27)</a>:</h4>
<p>Oh, interesting. Probably <code>insert</code> isn't checking for non-emptiness before pushing an empty vector on the list. Lemme add that just so that things are tasteful (it should be almost no cost, as <code>merge()</code> is cheap to merge an empty list).</p>



<a name="126791475"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791475" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791475">(May 19 2018 at 10:27)</a>:</h4>
<p>(I don't mind being praised in public by my idols, but don't feel obligated to do it :p — joking, of course :)</p>



<a name="126791527"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126791527" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126791527">(May 19 2018 at 10:29)</a>:</h4>
<blockquote>
<p>It seems like you'd also want a <code>From</code> for <code>Vec&lt;Tuple&gt;</code>, perhaps using specialization (?). For now, there is a <code>from_vec()</code> method, and using that gives some wins for me in the other project (much larger inputs relative to derived facts).</p>
</blockquote>
<p>Yeah this seems like a classic case for specialization</p>



<a name="126798897"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126798897" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126798897">(May 19 2018 at 15:21)</a>:</h4>
<p>Random further thoughts (I'll poke at things with time, but want to write them down): in several cases we have e.g.</p>
<div class="codehilite"><pre><span></span>        let subset_r1p = iteration.variable_indistinct(&quot;subset_r1p&quot;);
        let subset_r2p = iteration.variable_indistinct(&quot;subset_r2p&quot;);
        let subset_p = iteration.variable_indistinct(&quot;subset_p&quot;);
</pre></div>


<p>where if the tuples are ordered <code>p_r1_r2</code> say, then the same sorted lists work for <code>_r1p</code> and <code>_p</code>. We just have to swing around the order to <code>_pr1</code> and then describe a trait <code>ProvidesKeyVal</code> or something that allows one to dance around using either key (<code>p</code> or <code>pr1</code>).</p>



<a name="126799083"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126799083" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126799083">(May 19 2018 at 15:29)</a>:</h4>
<p>Hrm. Maybe more complicated than I imagine, due to the difficulty of going from a <code>&amp;(X, Y, Z)</code> to a <code>&amp;(X, Y)</code>.</p>



<a name="126799748"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126799748" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126799748">(May 19 2018 at 15:56)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> possibly stupid question, should we actually have (some?) inputs as indistinct variables ?</p>



<a name="126799773"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126799773" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126799773">(May 19 2018 at 15:57)</a>:</h4>
<p>Totally could, but I don't think anything good happens. The indistinctness prevents comparisons of new data against existing data, but this doesn't happen for the input collections.</p>



<a name="126799825"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126799825" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126799825">(May 19 2018 at 15:58)</a>:</h4>
<p>oh only affecting tuple creations</p>



<a name="126799902"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126799902" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126799902">(May 19 2018 at 16:01)</a>:</h4>
<p>don't those variables pass through the regular mechanism at the beginning though ?</p>



<a name="126799948"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126799948" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126799948">(May 19 2018 at 16:02)</a>:</h4>
<p>eg fill one the Variable inputs (not Relations) and it'll transition from the different tuple storage during the .changed() rounds</p>



<a name="126800005"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800005" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800005">(May 19 2018 at 16:04)</a>:</h4>
<p>(esp for some of the bigger inputs — <em>if</em> what I'm saying makes sense)</p>



<a name="126800153"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800153" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800153">(May 19 2018 at 16:10)</a>:</h4>
<p>ofc even if it did gain a bit, it'd be more worthwhile to go for the 10x cfg equiv route</p>



<a name="126800207"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800207" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800207">(May 19 2018 at 16:13)</a>:</h4>
<p>They do pass through the reg mechanisms, but it's all meant to be zero cost for the first relation that comes in. That is, a relation is sorted and distinct, so you only do any work once you have two of them. The transition from <code>to_add</code> to <code>recent</code> to <code>stable</code> is just a pointer move, until you have conflicting relations.</p>



<a name="126800208"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800208" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800208">(May 19 2018 at 16:13)</a>:</h4>
<p>(trying indistinct out on <code>rla</code> and <code>cfg_edge_p</code>seemed to make a bit of improvement)</p>



<a name="126800210"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800210" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800210">(May 19 2018 at 16:13)</a>:</h4>
<p>Hrm. Interesting that it did anything! It's possible I've botched the logic somewhere.</p>



<a name="126800249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800249">(May 19 2018 at 16:14)</a>:</h4>
<p>could be noise as well :3 (but it seemed to have faster rounds than usual) just another thing we can note</p>



<a name="126800252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800252">(May 19 2018 at 16:14)</a>:</h4>
<p>oh</p>



<a name="126800259"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800259" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800259">(May 19 2018 at 16:15)</a>:</h4>
<p>Hrm. Yeah, double checking the <code>insert</code> and <code>changed</code> logic, the <em>intent</em> is that nothing should happen if there is just one relation that comes in.</p>



<a name="126800262"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800262" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800262">(May 19 2018 at 16:15)</a>:</h4>
<p>you might also have made those improvements earlier today, while the rev polonius used is from yesterday</p>



<a name="126800304"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800304" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800304">(May 19 2018 at 16:16)</a>:</h4>
<p>Some improvements landed, not sure. The ones I know landed seemed to have zero impact on polonius, but helped me out on some other analyses.</p>



<a name="126800306"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800306" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800306">(May 19 2018 at 16:16)</a>:</h4>
<p>I'll try with everything uptodate, but it's probably not that important anyway :)</p>



<a name="126800313"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800313" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800313">(May 19 2018 at 16:17)</a>:</h4>
<p>These thing can matter, if the logic is dodgy. :)  The 2x improvement I got in non-polonius compute was adding a <code>From</code> implementation for <code>Vec&lt;Tuple&gt;</code> rather than using the <code>IntoIterator</code> one (which drains the vec and then collects it).</p>



<a name="126800360"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800360" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800360">(May 19 2018 at 16:19)</a>:</h4>
<p><em>cough even without the 2x it'd be faster than the ASPLOS paper cough</em></p>



<a name="126800799"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800799" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800799">(May 19 2018 at 16:36)</a>:</h4>
<p>the indistinct could be noise for the couple % over yesterday's rev</p>



<a name="126800805"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800805" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800805">(May 19 2018 at 16:37)</a>:</h4>
<p>I did try just updating to today's and seemed overall still a wee bit faster than yersterday's anyway (that is without any other change than the new rev)</p>



<a name="126800939"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800939" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800939">(May 19 2018 at 16:42)</a>:</h4>
<p>Cool; I can take a peek. It certainly shouldn't hurt, I just don't understand why it helps either. :)</p>



<a name="126800944"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800944" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800944">(May 19 2018 at 16:43)</a>:</h4>
<p>it doesn't really anymore so it was surely just a fluke of this high variance machine :/</p>



<a name="126800948"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126800948" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126800948">(May 19 2018 at 16:43)</a>:</h4>
<p>which makes perfect sense that it wouldn't :)</p>



<a name="126801773"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126801773" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126801773">(May 19 2018 at 17:17)</a>:</h4>
<p>Minor nit on your blog post <span class="user-mention" data-user-id="116609">@Frank McSherry</span> , I think datalog <code>:-</code> for bindings rather than <code>&lt;-</code>. Otherwise, very impressive work =D</p>



<a name="126802957"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126802957" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126802957">(May 19 2018 at 18:04)</a>:</h4>
<p>(I think there are competing conventions)</p>



<a name="126803158"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126803158" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126803158">(May 19 2018 at 18:12)</a>:</h4>
<blockquote>
<p>either way; it seems fine to me to let e.g. <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> or others do some of it, to get a feel for how <span class="emoji emoji-1f438" title="frog">:frog:</span> works</p>
</blockquote>
<p>what needs to be done?</p>



<a name="126803160"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126803160" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126803160">(May 19 2018 at 18:12)</a>:</h4>
<p>sorry, ETOOMANYMESSAGES</p>



<a name="126803161"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126803161" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126803161">(May 19 2018 at 18:12)</a>:</h4>
<p>didn't read</p>



<a name="126803162"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126803162" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126803162">(May 19 2018 at 18:12)</a>:</h4>
<p>tried to read some and figured that a lot of stuff was already using a context which I lack</p>



<a name="126803166"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126803166" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126803166">(May 19 2018 at 18:13)</a>:</h4>
<p>probably you have been talking since a while about some stuff and I have no clue :P</p>



<a name="126803169"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126803169" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126803169">(May 19 2018 at 18:13)</a>:</h4>
<p>just figured that you changed datalog with datafrog, I still need to see what's that about</p>



<a name="126803213"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126803213" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126803213">(May 19 2018 at 18:14)</a>:</h4>
<p>classic... Sorry Frank</p>



<a name="126803223"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126803223" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126803223">(May 19 2018 at 18:15)</a>:</h4>
<p>I'm gonna spin off a new topic <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> for just this :)</p>



<a name="126821664"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126821664" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126821664">(May 20 2018 at 06:24)</a>:</h4>
<p>More random optimization thoughts (not sure how best to effect, without substantial changes): queries like </p>
<div class="codehilite"><pre><span></span>    // subset(R1, R2, Q) :-
    //   subset(R1, R2, P) :-
    //   cfg_edge(P, Q),
    //   region_live_at(R1, Q),
    //   region_live_at(R2, Q).
</pre></div>


<p>require a few collections that we maintain (subsets <code>_1</code> and <code>_2</code>). In fact, once we get going we don't need these collections, because none of the other relations change, so no one actually consults them at all. We need the tuples to be sorted and batched, but not maintained (distinctly or not). So the random merging and sorting of lists isn't really a big deal.</p>



<a name="126822154"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126822154" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126822154">(May 20 2018 at 06:44)</a>:</h4>
<p><code>subset_r2p</code> looks dead in datafrog_opt; can anyone confirm that? I get a small bump by deleting it.</p>



<a name="126826189"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126826189" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126826189">(May 20 2018 at 09:31)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> I don't quite understand the distinction you are drawing between "sorted/batched" and "maintained"</p>



<a name="126826521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126826521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126826521">(May 20 2018 at 09:45)</a>:</h4>
<p>(<code>subset_r2p</code> is unused indeed)</p>



<a name="126826616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126826616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126826616">(May 20 2018 at 09:49)</a>:</h4>
<p>(so is <code>live_to_dead_regions_p</code>)</p>



<a name="126826760"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126826760" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126826760">(May 20 2018 at 09:55)</a>:</h4>
<p>maybe the distinction is that we could need less data, as being only used as temporaries, only the round's recent batch is effectively used, while <span class="emoji emoji-1f438" title="frog">:frog:</span> will still assume we might want to .complete() it one day ?</p>



<a name="126826947"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126826947" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126826947">(May 20 2018 at 10:02)</a>:</h4>
<p>it's subtle. But I can see something along those lines, perhaps specifically because they are not distinct?</p>



<a name="126826951"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126826951" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126826951">(May 20 2018 at 10:02)</a>:</h4>
<p>or maybe I'm misremembering</p>



<a name="126826961"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126826961" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126826961">(May 20 2018 at 10:03)</a>:</h4>
<p>anyway I'm still having my morning coffee =) not fully awake yet</p>



<a name="126827062"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827062" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827062">(May 20 2018 at 10:07)</a>:</h4>
<p>(tbf so am I, but it's noon :p)</p>



<a name="126827105"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827105" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827105">(May 20 2018 at 10:09)</a>:</h4>
<p>PS a PR removing <code>subset_r2p</code> and <code>live_to_dead_regions_p</code> would not be unwelcome :)</p>



<a name="126827107"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827107" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827107">(May 20 2018 at 10:10)</a>:</h4>
<p>on it (and updating to frog's latest rev)</p>



<a name="126827350"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827350" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827350">(May 20 2018 at 10:19)</a>:</h4>
<blockquote>
<p>it's subtle. But I can see something along those lines, perhaps specifically because they are not distinct?</p>
</blockquote>
<p>to elaborate on what I mean: I think <span class="user-mention" data-user-id="116113">@lqd</span> you were suggesting that those variables don't <em>need</em> a "stable"? But I thought it was still used to sometimes catch duplicates — but I guess the point is that if there is some new tuple in <code>subset</code>, it will <em>always</em> be new in <code>subset_1</code> (which is <code>subset</code> join <code>cfg_edge</code>). That... it true here, I guess, and distinct has nothing to do with it.</p>
<p>(But I suppose that if the closure given to <code>from_join</code> were to drop some fields, it might not be true, in which case distinct would be important, right?)</p>



<a name="126827447"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827447" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827447">(May 20 2018 at 10:22)</a>:</h4>
<p>I thought it could be that — but I'm no Frank :D — and for this very specific case of indexing, where we wouldn't/shouldn't drop fields to get this benefit ?</p>



<a name="126827453"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827453" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827453">(May 20 2018 at 10:23)</a>:</h4>
<p>PR is up <a href="https://github.com/rust-lang-nursery/polonius/pull/40" target="_blank" title="https://github.com/rust-lang-nursery/polonius/pull/40">https://github.com/rust-lang-nursery/polonius/pull/40</a> (I haven't ran it a lot for a before/after, but the numbers seem lower than before, on this slow/high variance machine)</p>



<a name="126827541"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827541" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827541">(May 20 2018 at 10:26)</a>:</h4>
<p>yeah, I see some slight win, although the numbers are a bit jumpier today than yesterday for some weird reason</p>



<a name="126827686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827686">(May 20 2018 at 10:32)</a>:</h4>
<p>Good news (esp for <span class="user-mention" data-user-id="116113">@lqd</span> ): I implemented leapfrog triejoin.</p>



<a name="126827688"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827688" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827688">(May 20 2018 at 10:32)</a>:</h4>
<p>Or, leapfrog join, let's say. No tries involved yet.</p>



<a name="126827690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827690">(May 20 2018 at 10:32)</a>:</h4>
<p>:D</p>



<a name="126827696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827696">(May 20 2018 at 10:33)</a>:</h4>
<p>I've got a few changes in place, but they are about a 1.5s improvement over what I had before. Still poking at things though.</p>



<a name="126827735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827735">(May 20 2018 at 10:34)</a>:</h4>
<p>Main gist is that when doing multiple joins to extend just one attribute, you can do it all at once and save yourself the pain of intermediate materialization:</p>
<div class="codehilite"><pre><span></span><span class="w">            </span><span class="c1">// subset(R1, R2, Q) :-</span>
<span class="w">            </span><span class="c1">//   subset(R1, R2, P) :-</span>
<span class="w">            </span><span class="c1">//   cfg_edge(P, Q),</span>
<span class="w">            </span><span class="c1">//   region_live_at(R1, Q),</span>
<span class="w">            </span><span class="c1">//   region_live_at(R2, Q).</span>
<span class="w">            </span><span class="c1">//</span>
<span class="w">            </span><span class="c1">// Carry `R1 &lt;= R2` from P into Q if both `R1` and</span>
<span class="w">            </span><span class="c1">// `R2` are live in Q.</span>
<span class="w">            </span><span class="k">use</span><span class="w"> </span><span class="n">datafrog</span>::<span class="n">leapfrog</span>::<span class="p">{</span><span class="n">LeapFrog</span><span class="p">,</span><span class="w"> </span><span class="n">Wrapper</span><span class="p">,</span><span class="w"> </span><span class="n">leapfrog_into</span><span class="p">};</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Wrapper</span>::<span class="n">from</span><span class="p">(</span><span class="o">&amp;</span><span class="n">cfg_edge_rel</span><span class="p">,</span><span class="w">       </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">p</span><span class="p">,(</span><span class="n">_r1</span><span class="p">,</span><span class="n">_r2</span><span class="p">))</span><span class="o">|</span><span class="w"> </span><span class="n">p</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Wrapper</span>::<span class="n">from</span><span class="p">(</span><span class="o">&amp;</span><span class="n">region_live_at_rel</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">_p</span><span class="p">,(</span><span class="n">r1</span><span class="p">,</span><span class="n">_r2</span><span class="p">))</span><span class="o">|</span><span class="w"> </span><span class="n">r1</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper3</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Wrapper</span>::<span class="n">from</span><span class="p">(</span><span class="o">&amp;</span><span class="n">region_live_at_rel</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">_p</span><span class="p">,(</span><span class="n">_r1</span><span class="p">,</span><span class="n">r2</span><span class="p">))</span><span class="o">|</span><span class="w"> </span><span class="n">r2</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">leapers</span>: <span class="p">[</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">LeapFrog</span><span class="o">&lt;</span><span class="n">_</span><span class="p">,</span><span class="n">_</span><span class="o">&gt;</span><span class="p">;</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper1</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper2</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper3</span><span class="p">];</span><span class="w"></span>
<span class="w">            </span><span class="n">leapfrog_into</span><span class="p">(</span><span class="o">&amp;</span><span class="n">subset_p</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">leapers</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="n">subset_p</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">_p</span><span class="p">,(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">)),</span><span class="o">&amp;</span><span class="n">q</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">q</span><span class="p">,(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">)));</span><span class="w"></span>
</pre></div>



<a name="126827736"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827736" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827736">(May 20 2018 at 10:34)</a>:</h4>
<p>this work has both scientific and comedic value</p>



<a name="126827743"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827743" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827743">(May 20 2018 at 10:35)</a>:</h4>
<p>You just need to indicate for each of the participating relations how they should search for a tuple prefix, which clues them in on how to propose/reject possible extension values.</p>



<a name="126827786"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827786" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827786">(May 20 2018 at 10:36)</a>:</h4>
<p>Leaper::from</p>



<a name="126827792"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827792" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827792">(May 20 2018 at 10:37)</a>:</h4>
<p>Interesting. We'll need a name for the antijoin variant (currently <code>AntiWrapper</code>) that leaps around avoiding things. <code>Frogger</code>, perhaps?</p>



<a name="126827833"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827833" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827833">(May 20 2018 at 10:38)</a>:</h4>
<p>E.g.</p>
<div class="codehilite"><pre><span></span><span class="w">            </span><span class="c1">// live_to_dead_regions(R1, R2, P, Q) :-</span>
<span class="w">            </span><span class="c1">//   subset(R1, R2, P),</span>
<span class="w">            </span><span class="c1">//   cfg_edge(P, Q),</span>
<span class="w">            </span><span class="c1">//   region_live_at(R1, Q),</span>
<span class="w">            </span><span class="c1">//   !region_live_at(R2, Q).</span>

<span class="w">            </span><span class="k">use</span><span class="w"> </span><span class="n">datafrog</span>::<span class="n">leapfrog</span>::<span class="p">{</span><span class="n">LeapFrog</span><span class="p">,</span><span class="w"> </span><span class="n">Wrapper</span><span class="p">,</span><span class="w"> </span><span class="n">AntiWrapper</span><span class="p">,</span><span class="w"> </span><span class="n">leapfrog_into</span><span class="p">};</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Wrapper</span>::<span class="n">from</span><span class="p">(</span><span class="o">&amp;</span><span class="n">cfg_edge_rel</span><span class="p">,</span><span class="w">       </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">p</span><span class="p">,(</span><span class="n">_r1</span><span class="p">,</span><span class="n">_r2</span><span class="p">))</span><span class="o">|</span><span class="w"> </span><span class="n">p</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Wrapper</span>::<span class="n">from</span><span class="p">(</span><span class="o">&amp;</span><span class="n">region_live_at_rel</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">_p</span><span class="p">,(</span><span class="n">r1</span><span class="p">,</span><span class="n">_r2</span><span class="p">))</span><span class="o">|</span><span class="w"> </span><span class="n">r1</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper3</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">AntiWrapper</span>::<span class="n">from</span><span class="p">(</span><span class="o">&amp;</span><span class="n">region_live_at_rel</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">_p</span><span class="p">,(</span><span class="n">_r1</span><span class="p">,</span><span class="n">r2</span><span class="p">))</span><span class="o">|</span><span class="w"> </span><span class="n">r2</span><span class="p">);</span><span class="w"></span>
<span class="w">            </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">leapers</span>: <span class="p">[</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">LeapFrog</span><span class="o">&lt;</span><span class="n">_</span><span class="p">,</span><span class="n">_</span><span class="o">&gt;</span><span class="p">;</span><span class="w"> </span><span class="mi">3</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper1</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper2</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">wrapper3</span><span class="p">];</span><span class="w"></span>
<span class="w">            </span><span class="n">leapfrog_into</span><span class="p">(</span><span class="o">&amp;</span><span class="n">subset_p</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">leapers</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="n">live_to_dead_regions</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">p</span><span class="p">,(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">)),</span><span class="o">&amp;</span><span class="n">q</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">,</span><span class="n">p</span><span class="p">,</span><span class="n">q</span><span class="p">));</span><span class="w"></span>
</pre></div>



<a name="126827834"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827834" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827834">(May 20 2018 at 10:38)</a>:</h4>
<p>(I feel niko wanted to have the bad guy from the muppets movie in there somewhere)</p>



<a name="126827840"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827840" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827840">(May 20 2018 at 10:39)</a>:</h4>
<p>Doc Hopper?</p>



<a name="126827881"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827881" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827881">(May 20 2018 at 10:40)</a>:</h4>
<p>Constantine ? <a href="#narrow/stream/122657-wg-nll/subject/porting-to-datafrog/near/126803992" title="#narrow/stream/122657-wg-nll/subject/porting-to-datafrog/near/126803992">https://rust-lang.zulipchat.com/#narrow/stream/122657-wg-nll/subject/porting-to-datafrog/near/126803992</a> (I know very little about the topic sorry :p)</p>



<a name="126827935"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827935" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827935">(May 20 2018 at 10:43)</a>:</h4>
<p>in any case, super good news</p>



<a name="126827977"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126827977" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126827977">(May 20 2018 at 10:44)</a>:</h4>
<p>It's all a bit hack-y at the moment. May take a day or two to tidy this up. I don't have a sense for whether there is a time crunch on anything here. Perhaps eventually good to push a crate, once the types settled down a bit.</p>



<a name="126828144"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126828144" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126828144">(May 20 2018 at 10:52)</a>:</h4>
<p>maybe the concept of parts/steps/hops/leap</p>
<div class="codehilite"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">cfg_edge_leap</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cfg_edge_rel</span><span class="p">.</span><span class="n">join_leap</span><span class="p">(</span><span class="o">|&amp;</span><span class="p">(</span><span class="n">p</span><span class="p">,(</span><span class="n">_r1</span><span class="p">,</span><span class="n">_r2</span><span class="p">))</span><span class="o">|</span><span class="w"> </span><span class="n">p</span><span class="p">);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">region_live_at_leap</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">region_live_at_rel</span><span class="p">.</span><span class="n">join_leap</span><span class="p">(</span><span class="o">&amp;</span><span class="p">(</span><span class="n">_p</span><span class="p">,(</span><span class="n">r1</span><span class="p">,</span><span class="n">_r2</span><span class="p">))</span><span class="o">|</span><span class="w"> </span><span class="n">r1</span><span class="p">);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">region_live_at_anti_leap</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">region_live_at_rel</span><span class="p">.</span><span class="n">antijoin_leap</span><span class="p">(</span><span class="o">|&amp;</span><span class="p">(</span><span class="n">_p</span><span class="p">,(</span><span class="n">_r1</span><span class="p">,</span><span class="n">r2</span><span class="p">))</span><span class="o">|</span><span class="w"> </span><span class="n">r2</span><span class="p">);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">leaps</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">cfg_edge_leap</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">region_live_at_leap</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">region_live_at_anti_leap</span><span class="p">];</span><span class="w"></span>
<span class="n">live_to_dead_regions</span><span class="p">.</span><span class="n">from_leaps</span><span class="p">(</span><span class="o">&amp;</span><span class="n">subset_p</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">leaps</span><span class="p">,</span><span class="w"> </span><span class="o">|&amp;</span><span class="p">(</span><span class="n">p</span><span class="p">,(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">)),</span><span class="o">&amp;</span><span class="n">q</span><span class="o">|</span><span class="w"> </span><span class="p">(</span><span class="n">r1</span><span class="p">,</span><span class="n">r2</span><span class="p">,</span><span class="n">p</span><span class="p">,</span><span class="n">q</span><span class="p">));</span><span class="w"></span>
</pre></div>



<a name="126828449"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126828449" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126828449">(May 20 2018 at 11:02)</a>:</h4>
<p>Also, it looks like <code>requires</code> is only ever used followed by a <code>!killed</code>, which makes me think we could put it as part of the <code>requires</code> derivation. I.e. make <code>requires</code> into <code>requires_but_not_killed</code> or something like that. Save all the antijoins with killed (which is not large, but which currently results in materializations).</p>
<p>Off to explore a bit! Will check back later.</p>



<a name="126828453"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126828453" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126828453">(May 20 2018 at 11:02)</a>:</h4>
<p>(or ofc <code>extend_from_leaps</code>)</p>



<a name="126836161"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126836161" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126836161">(May 20 2018 at 16:01)</a>:</h4>
<p>So I've been playing with trying to build a more efficient implementation of <code>Relation::merge</code> that actually zips the two tuple lists instead of just appending them and then doing a sort/dedup</p>



<a name="126836202"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126836202" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126836202">(May 20 2018 at 16:02)</a>:</h4>
<p>I'm not sure I'll be able to make it faster, but one interesting thing I noticed is that there are a not-insignificant number of cases where one of the tuple lists basically just picks up after the other one leaves off<br>
i.e. <code>elements1[elements1.len() - 1] &lt; elements2[0]</code></p>



<a name="126836212"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126836212" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126836212">(May 20 2018 at 16:03)</a>:</h4>
<p>not sure why exactly, but that "fast path" might be worth keeping even if my other stuff doesn't work out</p>



<a name="126836312"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126836312" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126836312">(May 20 2018 at 16:07)</a>:</h4>
<blockquote>
<p>Good news (esp for <span class="user-mention" data-user-id="116113">@lqd</span> ): I implemented leapfrog triejoin.</p>
</blockquote>
<p>clearly this should be <code>triefrog</code>. <code>leaptriefrog</code>?</p>



<a name="126836449"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126836449" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126836449">(May 20 2018 at 16:12)</a>:</h4>
<p>unfortunately there are no tries ATM, only joins :3 &lt;insert dog logic meme here&gt;</p>



<a name="126836773"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126836773" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126836773">(May 20 2018 at 16:25)</a>:</h4>
<p><span class="user-mention" data-user-id="116106">@Reed Koser</span> That's interesting! (the continuation thing). I was a bit lazy here, relying on the fact that <code>sort()</code> does a merge sort, and so it should i. identify the sorted regions, then ii. do a merge as fast as Rust knows how. In principle this can be sped up (especially when we want to do multiple merges), but it may not be too far off of the "right answer".</p>
<p>Another random observation: the join orders and tuple orders and stuff have implications for sorting. There are times where the tuple layout is different, but the order is the same or very similar (e.g. flipping the last two elements). Paying attention to these things might make sorting less painful (though I didn't pay much attention myself).</p>



<a name="126836922"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126836922" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126836922">(May 20 2018 at 16:30)</a>:</h4>
<p><span class="emoji emoji-1f44d" title="+1">:+1:</span> I haven't looked in to how map etc. maintain the sorted invariants. Initial experiments seem to indicate that my zipper merge is a little faster, but I'm currently playing whack-a-mole with system issues (curse you dynamic frequency scaling!) to make sure my benchmarks are testing what I think they are</p>



<a name="126836925"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126836925" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126836925">(May 20 2018 at 16:31)</a>:</h4>
<p>Another way to speed up the computation, perhaps, is to observe that all of the <code>Loan</code> computations are independent (there are no rules that change the loan associated with the tuple). This means that one could implement everything as an outer loop over borrows, with the inner component of each loop having a much smaller working set. It's not obviously a good fit for <span class="emoji emoji-1f438" title="frog">:frog:</span> which prefers to do everything at the same time, but worth keeping in mind (also, perhaps more obvious when one could stop early: a loan requirement can't cross cfg edges where it is killed, independent of the number of regions alive there).</p>



<a name="126837019"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837019" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837019">(May 20 2018 at 16:34)</a>:</h4>
<p><span class="user-mention" data-user-id="116106">@Reed Koser</span> Another related optimization: the <code>self.stable</code> field could probably be one <code>Vec&lt;Tuple&gt;</code> which has delineated regions that are sorted. This means that rather than lots of merges, one just removes delineations and sorts (merges) regions. This saves the allocations and memcpys and such, which .. are some fraction of the work that is going on.</p>



<a name="126837066"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837066" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837066">(May 20 2018 at 16:36)</a>:</h4>
<p>Also,  though perhaps this isn't the best time (boarding!) I'd love to get a rundown of the intent of the _opt variants. Clearly the <code>subset</code> and <code>requires</code> they compute are different, and I'd love to know how and why (are there better names for them, for someone starting from the naive implementation?).</p>



<a name="126837068"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837068" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837068">(May 20 2018 at 16:37)</a>:</h4>
<p>I can look in to that once I'm sure my merge is faster<br>
One disadvantage of the approach I'm taking is that it always allocates. I could probably play some clever indexing tricks and shuffle things around in memory  to make it work, but I don't think it's possible to do what I want without either allocating or using lots of unsafe</p>



<a name="126837218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837218">(May 20 2018 at 16:42)</a>:</h4>
<p>IIRC the opt was started mainly to limit the TC over <code>subset</code> which Frank had warned against doing :)</p>



<a name="126837226"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837226" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837226">(May 20 2018 at 16:43)</a>:</h4>
<p>at first, then also <code>requires</code>(edited with what I actually meant to say)</p>



<a name="126837265"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837265" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837265">(May 20 2018 at 16:44)</a>:</h4>
<p>yes we can see some of the steps Niko took in the PR  <a href="https://github.com/rust-lang-nursery/polonius/pull/23/commits" target="_blank" title="https://github.com/rust-lang-nursery/polonius/pull/23/commits">https://github.com/rust-lang-nursery/polonius/pull/23/commits</a></p>



<a name="126837272"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837272" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837272">(May 20 2018 at 16:45)</a>:</h4>
<p>yeah, I think eventually we wanted to drive computation of the subset relation from the invalidates facts</p>



<a name="126837316"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837316" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837316">(May 20 2018 at 16:46)</a>:</h4>
<p>I think <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> 's PRs are the first steps in that direction (<a href="https://github.com/rust-lang-nursery/polonius/pull/28" target="_blank" title="https://github.com/rust-lang-nursery/polonius/pull/28">https://github.com/rust-lang-nursery/polonius/pull/28</a> and <a href="https://github.com/rust-lang-nursery/polonius/pull/34" target="_blank" title="https://github.com/rust-lang-nursery/polonius/pull/34">https://github.com/rust-lang-nursery/polonius/pull/34</a>)</p>



<a name="126837317"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837317" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837317">(May 20 2018 at 16:46)</a>:</h4>
<p>Other random observations: even in frog-opt, about 0.5M facts in <code>subset</code> are symmetric relation facts, e.g. <code>(p, (r, r))</code>.</p>



<a name="126837327"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837327" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837327">(May 20 2018 at 16:47)</a>:</h4>
<p>These are probably logically redundant somehow. Trying to map out how many distinct equivalence classes of region there are.</p>



<a name="126837428"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126837428" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126837428">(May 20 2018 at 16:50)</a>:</h4>
<p>maybe Yannakakis' algorithm could provide us with a simple plan wrt join orders ? eg at the beginning of coding a computation, we'd describe the variables and needed joins, and "something" could give us some of this info, eg iteration.explain_analyze() ;)</p>



<a name="126838632"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126838632" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126838632">(May 20 2018 at 17:31)</a>:</h4>
<div class="codehilite"><pre><span></span>datafrog_opt            time:   [1.2379 ms 1.2448 ms 1.2527 ms]
                        change: [-27.409% -23.825% -20.116%] (p = 0.00 &lt; 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  4 (4.00%) high mild
  6 (6.00%) high severe
</pre></div>


<p><span class="emoji emoji-1f44c" title="ok hand">:ok_hand:</span> <span class="emoji emoji-1f438" title="frog">:frog:</span></p>



<a name="126838671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126838671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126838671">(May 20 2018 at 17:32)</a>:</h4>
<p>I /think/ I measured it correctly</p>



<a name="126838823"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126838823" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126838823">(May 20 2018 at 17:39)</a>:</h4>
<p><a href="https://github.com/frankmcsherry/datafrog/pull/2" target="_blank" title="https://github.com/frankmcsherry/datafrog/pull/2">https://github.com/frankmcsherry/datafrog/pull/2</a></p>



<a name="126839030"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126839030" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126839030">(May 20 2018 at 17:47)</a>:</h4>
<p><span class="user-mention" data-user-id="116106">@Reed Koser</span> Dunno if it applies, but could maybe use <a href="https://docs.rs/itertools/0.7.8/itertools/trait.Itertools.html#method.merge" target="_blank" title="https://docs.rs/itertools/0.7.8/itertools/trait.Itertools.html#method.merge">https://docs.rs/itertools/0.7.8/itertools/trait.Itertools.html#method.merge</a> ?</p>



<a name="126839075"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126839075" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126839075">(May 20 2018 at 17:48)</a>:</h4>
<p>No, it's not just a simple merge. We also need to de-duplicate at the same time</p>



<a name="126839085"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126839085" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126839085">(May 20 2018 at 17:49)</a>:</h4>
<p>I mean we don't /need/ to but we can and I think it's faster</p>



<a name="126839086"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126839086" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126839086">(May 20 2018 at 17:49)</a>:</h4>
<p>Haven't tested it though so who knows</p>



<a name="126839196"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126839196" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126839196">(May 20 2018 at 17:53)</a>:</h4>
<p>(I had mentioned the empty rels I saw in merge before, but in my limited testing special casing them didn't make a big difference) the zipper merge does improve datafrog_opt on clap indeed, sweet <span class="emoji emoji-1f44d" title="thumbs up">:thumbs_up:</span></p>



<a name="126839746"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126839746" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126839746">(May 20 2018 at 18:10)</a>:</h4>
<p>Nice, I'm glad my benchmarking did actually mean something =P</p>



<a name="126849850"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126849850" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126849850">(May 21 2018 at 00:29)</a>:</h4>
<p>So, I've spent some time profiling <code>polonius</code></p>



<a name="126849903"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126849903" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126849903">(May 21 2018 at 00:31)</a>:</h4>
<p>It seems like <code>Relation::merge</code> is still the largest contributor to runtime, followed by <code>gallop</code> (as part of <code>join_core</code>, presumably)</p>



<a name="126849911"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126849911" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126849911">(May 21 2018 at 00:31)</a>:</h4>
<p>which I guess is unsurprising</p>



<a name="126849914"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126849914" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126849914">(May 21 2018 at 00:31)</a>:</h4>
<p>I'm trying to get data about which relations/operations are the worst offenders now</p>



<a name="126852675"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126852675" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126852675">(May 21 2018 at 02:16)</a>:</h4>
<p><a href="/user_uploads/4715/oXYHV6R4XE8eRkNf-w6C3hP3/pasted_image.png" target="_blank" title="pasted_image.png">pasted image</a> I started making graphs again <span class="emoji emoji-1f609" title="wink">:wink:</span></p>
<div class="message_inline_image"><a href="/user_uploads/4715/oXYHV6R4XE8eRkNf-w6C3hP3/pasted_image.png" target="_blank" title="pasted image"><img src="/user_uploads/4715/oXYHV6R4XE8eRkNf-w6C3hP3/pasted_image.png"></a></div>



<a name="126854519"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126854519" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126854519">(May 21 2018 at 03:26)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span>  did you ever play with using some sort of hash-based join instead of the sort-and-zip approach? I feel like it might help perf since instead of maintaining a bunch of independent "views" of the data (<code>subset_r1p</code>, <code>subset_r2p</code>, etc.), we can keep a single "source of truth" vector and have the joins accelerated by something like <code>HashMap&lt;K, usize&gt;</code> maps. Right now it seems like tuples flow through the system, sometimes with a latency of like 3 rounds, before finally coming back around to impact the same set that kicked off their insertion in the first place. So you get <a href="/user_uploads/4715/kJaaOty_FZCm2cr2bc8Vd0vH/pasted_image.png" target="_blank" title="pasted_image.png">this kind of rippling effect</a> though all of the variables as changes express themselves after some latency. I know we'll have to accept some of that, but from what I've seen in the profiling data the best thing to do to improve perf seems to be reducing the number of variables (and maybe rounds?) that we have to do merges on.</p>
<div class="message_inline_image"><a href="/user_uploads/4715/kJaaOty_FZCm2cr2bc8Vd0vH/pasted_image.png" target="_blank" title="this kind of rippling effect"><img src="/user_uploads/4715/kJaaOty_FZCm2cr2bc8Vd0vH/pasted_image.png"></a></div>



<a name="126854613"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126854613" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126854613">(May 21 2018 at 03:30)</a>:</h4>
<p>It might end up all coming out in the wash though, since to support my vision of hash-based joins we still need to maintain a O(N) sized dictionary of key-&gt;indices plus we would have garbage cache locality on accesses to the datastore in all likelyhood</p>



<a name="126854614"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126854614" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126854614">(May 21 2018 at 03:30)</a>:</h4>
<p>/me is like 30% of the way through implementing this so I guess we'll see</p>



<a name="126856942"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126856942" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126856942">(May 21 2018 at 05:03)</a>:</h4>
<p>... and I think that's about as far as I'm going to get. The ownership issues are many, and I'd either need to do a boatload of <code>Rc</code> based pointer chasing or train cars full of copies to get it compiling. Either way memory usage would probably be much higher for very little perf gain. It was educational, if not necessarily a step in the right direction.</p>



<a name="126856991"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126856991" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126856991">(May 21 2018 at 05:05)</a>:</h4>
<p>and the borrow checker really saved me here, if it was C++ I would have gotten really sad after chasing iterator invalidation bugs forever =)</p>



<a name="126860891"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126860891" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126860891">(May 21 2018 at 07:29)</a>:</h4>
<p>Looming updates (on my laptop):</p>
<p>1. Pointing polonius at new datafrog master:  7.680s -&gt; 6.934s.<br>
2. Using treefrog leapjoin: 6.934s -&gt; 5.744s.</p>
<p>The treefrog leapjoin (lftj is logicblox's property) structures some joins differently, and it would be good to explain this so that folks can take advantage. Roughly, the more constraints you can put in at the same time the better, and the more you break apart queries into little relations the less better. I'll get a blog post out today I suspect.</p>
<p>Also, with current numbers, on clap the <code>subset</code> relation is larger than all the others put together. It is also a great candidate for TFLJing, in that it is mostly a set of rules for how to extend <code>requires</code>, rather than anything we need to materialize. That possibility might be speculation on my part, so I'll try and explain how inlining <code>subset</code> could help come blogpost time.</p>



<a name="126863517"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126863517" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126863517">(May 21 2018 at 09:01)</a>:</h4>
<p>(I was looking at reducing the <code>cfg</code>relation, nothing particular to report at this time; apart from the exact idea mentioned in issue #20 I think, will not help for clap itself)</p>



<a name="126863868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126863868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126863868">(May 21 2018 at 09:12)</a>:</h4>
<p>Continued:</p>
<p>3. Write own <code>binary_search</code> method: 5.744s -&gt; 5.335s</p>



<a name="126864157"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126864157" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126864157">(May 21 2018 at 09:22)</a>:</h4>
<p>maybe a different sorting algorithm</p>



<a name="126864963"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126864963" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126864963">(May 21 2018 at 09:45)</a>:</h4>
<p>Maybe! Though, it's not so much the performance of Rust's <code>binary_search</code> as the guarantees it does/doesn't provide.</p>



<a name="126865007"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126865007" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126865007">(May 21 2018 at 09:46)</a>:</h4>
<p>It is important to have a binary search that finds the first match of a thing (e.g. a key, in a list of (key,val)), so that we can scan forward and find all other matches.</p>



<a name="126865011"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126865011" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126865011">(May 21 2018 at 09:46)</a>:</h4>
<p>I think the current implementation does this, but the team declined to commit to it doing this.</p>



<a name="126867121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126867121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126867121">(May 21 2018 at 10:57)</a>:</h4>
<p><a href="https://github.com/frankmcsherry/blog/blob/master/posts/2018-05-19.md#addendum-2018-05-21-treefrog-leapjoin" target="_blank" title="https://github.com/frankmcsherry/blog/blob/master/posts/2018-05-19.md#addendum-2018-05-21-treefrog-leapjoin">https://github.com/frankmcsherry/blog/blob/master/posts/2018-05-19.md#addendum-2018-05-21-treefrog-leapjoin</a></p>



<a name="126867355"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126867355" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126867355">(May 21 2018 at 11:04)</a>:</h4>
<p>Associated with this, I have a highly modified <code>datafrog_opt.rs</code> that might want a different name to land. It should be the same logic, but it is different enough that perhaps it makes sense to keep both versions around for the moment?</p>



<a name="126870351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126870351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126870351">(May 21 2018 at 12:39)</a>:</h4>
<blockquote>
<p>Other random observations: even in frog-opt, about 0.5M facts in <code>subset</code> are symmetric relation facts, e.g. <code>(p, (r, r))</code>.</p>
</blockquote>
<p>wait, are they <em>literally</em> <code>r &lt;= r</code>? then we really don't need them...</p>



<a name="126870399"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126870399" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126870399">(May 21 2018 at 12:40)</a>:</h4>
<blockquote>
<p>Also,  though perhaps this isn't the best time (boarding!) I'd love to get a rundown of the intent of the _opt variants. Clearly the <code>subset</code> and <code>requires</code> they compute are different, and I'd love to know how and why (are there better names for them, for someone starting from the naive implementation?).</p>
</blockquote>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> basically my intution was to avoid computing the transitive closure, and instead only propagate edges where needed. The purpose of computing the transitive closure, for the most part, was that sometimes we would remove intermediate regions but we wanted to keep the overall effect. Example:</p>
<p>At some point P, you add <code>r1 &lt;= r2</code> and <code>r2 &lt;= r3</code>. Then in the successor point <code>Q</code>, <code>r2</code> goes dead. We still want to ensure that (at Q) we know that <code>r1 &lt;= r3</code>. </p>
<p>We used to compute TC and then remove everything related to <code>r2</code>. We now do this more lazilly, only computing the reachability for regions like <code>r2</code> that (A) are dead in Q but (B) have a live predecessor in Q.</p>



<a name="126897903"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126897903" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126897903">(May 21 2018 at 23:23)</a>:</h4>
<blockquote>
<p>Associated with this, I have a highly modified <code>datafrog_opt.rs</code> that might want a different name to land. It should be the same logic, but it is different enough that perhaps it makes sense to keep both versions around for the moment?</p>
</blockquote>
<p>if it's mostly about using the leapfrog join, it could also be good for someone to reimplement it and gain familiarity :)</p>



<a name="126910333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126910333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126910333">(May 22 2018 at 06:59)</a>:</h4>
<p>That sounds good too. I've got them locally now, but can gist them around or hand them off to anyone who wants to learn more. </p>
<p>Apropos that, I'm trying to grok the <code>_opt</code> rules to see if there are opportunities to use treefrog better. The main thing is that with this infrastructure, the actions that "cost" are adding a new variable; the large number of involved relations only improve the situation (by constraining the variable more). The "number of joins" is not especially important. With that in mind, it is possible that the <code>_opt</code> rules could be improved by fusing together a bunch of the rules, to try and get roughly one <code>leapjoin</code> per variable introduced. It may not be exactly that easy, but I'll try and fish out some examples to show what I mean.</p>



<a name="126910735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126910735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126910735">(May 22 2018 at 07:13)</a>:</h4>
<p>a bit like folding "subset" into "requires" like you mentioned earlier</p>



<a name="126910836"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126910836" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126910836">(May 22 2018 at 07:16)</a>:</h4>
<p>Yeah, if that would work out it would be great. I .. can't understand the <code>_opt</code> logic well enough to know if that will be the case, though. :)</p>



<a name="126910989"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126910989" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126910989">(May 22 2018 at 07:22)</a>:</h4>
<p>haha <span class="emoji emoji-1f642" title="simple smile">:simple_smile:</span> — I was rereading your "DD computation explanations" paper yesterday since I was about to track tuples flowing</p>



<a name="126911096"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126911096" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126911096">(May 22 2018 at 07:26)</a>:</h4>
<p>tbf this could rely on the limited datasets too much, might be prudent to wait for proper testing (or just focus on the rules but that's like, harder, man)</p>



<a name="126911252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126911252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126911252">(May 22 2018 at 07:32)</a>:</h4>
<p>(tbf2 the <code>invalidates</code> data could/should also cut down the cfg — in addition to another PR helping <code>subset</code> by reducing <code>outlives</code> by 50%)</p>



<a name="126911770"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126911770" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126911770">(May 22 2018 at 07:52)</a>:</h4>
<p>Yeah, I would be interested to see a bunch more inputs before specializing too much; that's a good point. Is the <code>clap</code> example the known-worst case, or is there a spectrum?</p>



<a name="126911772"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126911772" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126911772">(May 22 2018 at 07:52)</a>:</h4>
<p>I'll just chill for a bit then. ;)</p>



<a name="126911780"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126911780" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126911780">(May 22 2018 at 07:53)</a>:</h4>
<p>Btw, Datalog has simpler explanations than the DD explanations. There is a paper by <a href="https://yanniss.github.io" target="_blank" title="https://yanniss.github.io">https://yanniss.github.io</a> which has detail (and other good datalog links there).</p>



<a name="126911834"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126911834" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126911834">(May 22 2018 at 07:55)</a>:</h4>
<p>This is the explanations paper I'm thinking of: <a href="https://yanniss.github.io/DeclarativeDebugging.pdf" target="_blank" title="https://yanniss.github.io/DeclarativeDebugging.pdf">https://yanniss.github.io/DeclarativeDebugging.pdf</a></p>



<a name="126912587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126912587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126912587">(May 22 2018 at 08:21)</a>:</h4>
<p>thanks, I was also reading this one :)</p>



<a name="126912641"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126912641" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126912641">(May 22 2018 at 08:23)</a>:</h4>
<p>I was wondering the same about clap and worst-cases, it looks like to be one of the slow cases / the slowest case that is tracked on <a href="https://perf.rust-lang.org/" target="_blank" title="https://perf.rust-lang.org/">https://perf.rust-lang.org/</a> but wondered how "realistic" it was</p>



<a name="126912645"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126912645" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126912645">(May 22 2018 at 08:23)</a>:</h4>
<p>we can generate our own facts easily though</p>



<a name="126912689"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126912689" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126912689">(May 22 2018 at 08:24)</a>:</h4>
<p>which examples would be best, surely niko and felix know</p>



<a name="126914941"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126914941" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126914941">(May 22 2018 at 09:34)</a>:</h4>
<p>OK, I finally caught up with y'all on this TreeFrog LeapJoin business. =) Very cool!</p>



<a name="126918428"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126918428" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126918428">(May 22 2018 at 11:29)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> I think you said you wanted to keep #20 (condensing the CFG) in your backpocket for now, was this mostly because of the location::all issue ? or maybe also because of the current lack of tests/benchmarks, to validate if potential improvements are correct and not overfitted to the repo's couple datasets ?</p>



<a name="126920792"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126920792" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126920792">(May 22 2018 at 12:40)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> btw, in your leapjoin section about <code>dead_region_requires((R, P, Q), B)</code> I'm assuming you switched from joining with <code>requires_bp</code> like in the current version, to <code>requires_rp</code> because after leapjoining everything (let's call it, a leap surgery :3) you found out that you didn't need the <code>requires_bp</code> index anymore ? (not that it changes the results of course, just checking)</p>



<a name="126921044"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921044" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921044">(May 22 2018 at 12:48)</a>:</h4>
<p>Yeah, that is a good point: some of the performance improvement comes from many of the multiply-indexed relations simplifying down to just one relation. For example, there is only one copy of each of <code>subset</code> and <code>requires</code>.</p>



<a name="126921119"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921119" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921119">(May 22 2018 at 12:50)</a>:</h4>
<p>awesome thank you</p>



<a name="126921265"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921265" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921265">(May 22 2018 at 12:55)</a>:</h4>
<p><a href="https://gist.github.com/frankmcsherry/d3a5c56458779fa7df5934cbd68cb571" target="_blank" title="https://gist.github.com/frankmcsherry/d3a5c56458779fa7df5934cbd68cb571">https://gist.github.com/frankmcsherry/d3a5c56458779fa7df5934cbd68cb571</a></p>



<a name="126921268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921268">(May 22 2018 at 12:55)</a>:</h4>
<p>for the treefrogged version.</p>



<a name="126921419"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921419" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921419">(May 22 2018 at 12:59)</a>:</h4>
<p>I was trying it out with the Naive rules first :) which is nice</p>



<a name="126921690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921690">(May 22 2018 at 13:07)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> I was thinking as I rode the subway today: first off, I've not looked at your actual code, and I guess I have to review the blog post, but it seems like it would be useful if the <code>count</code> function provided some feedback to the function that filters tuples later on (e.g., where to start looking for matches etc)</p>



<a name="126921693"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921693" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921693">(May 22 2018 at 13:07)</a>:</h4>
<p>but secondly, we often have some information about ordering we could supply, though I'm not sure if that is useful</p>



<a name="126921783"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921783" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921783">(May 22 2018 at 13:09)</a>:</h4>
<p>I'm not sure I grok the sort of feedback that could be provided. Can you give an example? The <code>intersect</code> method picks up mostly where <code>count</code> leaves off, looking at the same range that it just went and checked the bounds of to get the count back.</p>



<a name="126921843"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921843" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921843">(May 22 2018 at 13:10)</a>:</h4>
<p>mm I guess a simple example would be that we know that <code>cfg_edge</code> will produce relatively matches</p>



<a name="126921856"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921856" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921856">(May 22 2018 at 13:11)</a>:</h4>
<p>so e.g. when I would try to optimize, I would try to order joins with <code>cfg_edge</code> earlier</p>



<a name="126921862"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921862" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921862">(May 22 2018 at 13:11)</a>:</h4>
<p>that said, it seems like the <code>count</code> mechanism is just more general and this may not be something we need to manually specify anymore</p>



<a name="126921866"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921866" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921866">(May 22 2018 at 13:11)</a>:</h4>
<p>which is even better</p>



<a name="126921882"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921882" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921882">(May 22 2018 at 13:12)</a>:</h4>
<p>I think there isn't much advantage to get here (unless I'm mis-understanding): the <code>count</code> method will return something like 1 pretty often (for <code>cfg</code>) and we will determine that it should be the one to propose the extension.</p>



<a name="126921927"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921927" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921927">(May 22 2018 at 13:12)</a>:</h4>
<p>right, I agree</p>



<a name="126921933"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921933" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921933">(May 22 2018 at 13:12)</a>:</h4>
<p>I guess my thought was that we might avoid invoking <code>count</code> — but then it's just doing work we kind of have to do anyway</p>



<a name="126921979"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126921979" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126921979">(May 22 2018 at 13:13)</a>:</h4>
<p>Yeah, <code>count</code> happens to id where we need to head in the big array of tuples anyhow (and that info is cached for <code>propose</code> and <code>intersect</code>). Even if we hard-wired in "always use <code>cfg</code> don't bother doing <code>count</code>" I think the times would be pretty similar. We could check though.</p>



<a name="126922060"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126922060" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126922060">(May 22 2018 at 13:15)</a>:</h4>
<p>nope, I'm convinced =)</p>



<a name="126965136"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126965136" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126965136">(May 23 2018 at 07:52)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> hello <span class="emoji emoji-1f44b" title="wave">:wave:</span> — just checking, even with leapfrog we couldn't  easily fold <code>subset</code> or <code>requires</code> right ? or <code>borrow_live_at</code> into the recent <code>errors(B, P) :- invalidates(P, B), borrow_live_at(B, P).</code> because all of those end up using more than one dynamic variable instead of just Relations. (I'm guessing we somehow can make a specific version of <code>borrow_live_at</code> that contains the <code>invalidates</code> join, for the regular mode; while in verbose mode we "have" to produce those triples; and I guess it might also change depending on how much of this error computation etc can be done in polonius rather than rustc)</p>



<a name="126965559"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126965559" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126965559">(May 23 2018 at 08:06)</a>:</h4>
<p>That sounds right. It isn't too hard to treefrog together two variables, but it isn't supported right now. You would need to i. write wrapper for Variables, which would need to check each of the relations when they count/propose/intersect, and ii. make sure that you write each of the update leapjoins (i.e. a leapjoin stemming from each of the involved variables, as each one just responds to changes in one relation).</p>



<a name="126965649"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126965649" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126965649">(May 23 2018 at 08:09)</a>:</h4>
<p>oh cool :) Maybe it can be a "good first issue" on the datafrog repo for people to contribute</p>



<a name="126965652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126965652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126965652">(May 23 2018 at 08:09)</a>:</h4>
<p>so that you don't have to do <em>all</em> the work heh</p>



<a name="126967689"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126967689" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126967689">(May 23 2018 at 09:08)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> </p>
<blockquote>
<p>while in verbose mode we "have" to produce those triples; and I guess it might also change depending on how much of this error computation etc can be done in polonius rather than rustc</p>
</blockquote>
<p>I think we can just drop "verbose" mode if it's getting in the way of something.</p>



<a name="126967756"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126967756" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126967756">(May 23 2018 at 09:10)</a>:</h4>
<p>totally; until we know exactly what rustc will require between the errors and the <code>borrow_live_at</code> tuples, this can be in flux for a while</p>



<a name="126970151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126970151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126970151">(May 23 2018 at 10:14)</a>:</h4>
<p>(nothing to be worried about anyway: even the multi variable join we know can be done would just be a bonus; leapfrog producing errors (but not <code>borrow_live_at</code>) is slightly faster than leapfrog producing <code>borrow_live_at</code>, we can even produce both by checking verbose mode during the iteration, and it still approximately the same — and both of those 2 are already faster than without leapfrog, like 20–30%; that being said I would also like to make sure the <code>errors</code>are actually correct otherwise those numbers are not that useful :) so I'll finish the frontend soon; meanwhile frank can rest and land this new API at his leisure :)</p>



<a name="126970341"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/126970341" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#126970341">(May 23 2018 at 10:21)</a>:</h4>
<p><span class="user-mention" data-user-id="116113">@lqd</span> ideally yeah, we trick someone most Rust-focused into coming up to speed on how <span class="emoji emoji-1f438" title="frog">:frog:</span> works, so that its understanding does live only in my head, and the code doesn't have the risk of rotting due to it not being clear how to make it do a specific thing. Such issues could be good, esp if you all id such a person.</p>



<a name="127141172"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/127141172" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Reed Koser <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#127141172">(May 26 2018 at 21:33)</a>:</h4>
<p>I took a stab at writing a merge that could handle multiple relations at once. It seems like it's generally slower, but here's the code <a href="https://github.com/rust-lang-nursery/datafrog/compare/master...bobtwinkles:multimerge?expand=1" target="_blank" title="https://github.com/rust-lang-nursery/datafrog/compare/master...bobtwinkles:multimerge?expand=1">https://github.com/rust-lang-nursery/datafrog/compare/master...bobtwinkles:multimerge?expand=1</a> so people can learn from my mistakes =P<br>
I'm just creating iterators over all the relations up front, and then stepping through them to generated a sorted list. Results (naming convention is merge_(# of relations)_(average relation size)/algorithm:</p>
<div class="codehilite"><pre><span></span>merge/merge_003x100/Individual
                        time:   [2.7191 us 2.7451 us 2.7767 us]
                        change: [-8.8104% -7.1505% -5.6703%] (p = 0.00 &lt; 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  7 (7.00%) high severe
merge/merge_003x100/MultiMerge
                        time:   [4.3975 us 4.4096 us 4.4266 us]
                        change: [-1.6184% -0.7284% +0.0416%] (p = 0.09 &gt; 0.05)
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  8 (8.00%) high severe

merge/merge_004x100000/Individual
                        time:   [7.0525 ms 7.0865 ms 7.1178 ms]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) low mild
merge/merge_004x100000/MultiMerge
                        time:   [7.9256 ms 7.9569 ms 7.9988 ms]
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe
</pre></div>


<p>My guess is that it's slower due to having poor cache locality (jumping all over memory to read from the different lists) compared to just merging things one at a time</p>



<a name="128772229"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128772229" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128772229">(Jun 28 2018 at 13:33)</a>:</h4>
<p>I was wondering whether using indexes would help doing the <span class="emoji emoji-1f438" title="frog">:frog:</span> joins, in addition to limiting the Variable indexes, or for such small relations maybe it wouldn't be that useful (maybe a compressed bitset, à la roaring bitmaps?)</p>



<a name="128772361"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128772361" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128772361">(Jun 28 2018 at 13:36)</a>:</h4>
<p>and also if <span class="user-mention" data-user-id="116609">@Frank McSherry</span> (hello ;) had thoughts about using a Factorized representation would be possible/useful/stupid (as IIRC there's also a worst-case optimal join algorithm that can work with them) and it seemed to lower both the space and time complexity (linked to the hypertree width) :)</p>



<a name="128772378"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128772378" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128772378">(Jun 28 2018 at 13:36)</a>:</h4>
<p>what kind of indexes are you referring to ?</p>



<a name="128772407"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128772407" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128772407">(Jun 28 2018 at 13:37)</a>:</h4>
<p>roaring bitmaps you mean ?</p>



<a name="128772415"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128772415" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128772415">(Jun 28 2018 at 13:37)</a>:</h4>
<p>or using them in the Variables themselves ?</p>



<a name="128772486"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128772486" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128772486">(Jun 28 2018 at 13:39)</a>:</h4>
<p>if the former: <a href="https://roaringbitmap.org/" target="_blank" title="https://roaringbitmap.org/">https://roaringbitmap.org/</a> a kind of hybrid structure</p>



<a name="128772801"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128772801" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128772801">(Jun 28 2018 at 13:45)</a>:</h4>
<p>(with tricks similar to map fusion, to enable the multiple chunks to use bitsets/arrays/etc independently)</p>



<a name="128773169"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128773169" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128773169">(Jun 28 2018 at 13:54)</a>:</h4>
<blockquote>
<p>roaring bitmaps you mean ?</p>
</blockquote>
<p>I've never heard of that either, but you wrote:</p>
<blockquote>
<p>...I was wondering whether using indexes would help..</p>
</blockquote>



<a name="128773177"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128773177" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128773177">(Jun 28 2018 at 13:55)</a>:</h4>
<p>I'm not sure what you meant by 'indexes' there</p>



<a name="128773179"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128773179" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128773179">(Jun 28 2018 at 13:55)</a>:</h4>
<p>I guess I have heard of roaring bitmaps</p>



<a name="128773247"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128773247" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128773247">(Jun 28 2018 at 13:56)</a>:</h4>
<p>we do use sparse bitsets in the compiler</p>



<a name="128773256"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128773256" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128773256">(Jun 28 2018 at 13:57)</a>:</h4>
<p>roaring's techniques feel potentially like overkill, but I'm not sure</p>



<a name="128773267"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128773267" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128773267">(Jun 28 2018 at 13:57)</a>:</h4>
<p>yeah, I was wondering how to compare using galloping vs computing the intersection of keys using indices and then joining the values</p>



<a name="128773418"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128773418" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128773418">(Jun 28 2018 at 14:00)</a>:</h4>
<p>and the other questions were about using factorized representations which seemed interesting to both reduce the number of materialized results, and the time to compute the join as well -- but maybe only on bigger datasets that we never reach with the NLL facts I'm not sure (and hoped Frank who knows everything would know whether this was a good idea before looking too much into it ;)</p>



<a name="128780592"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128780592" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128780592">(Jun 28 2018 at 16:30)</a>:</h4>
<p>So, the variables and galloping are pretty much a lightweight LSM index. They are throughput optimized rather than lookup optimized, so if you felt that there were perhaps many rounds producing few tuples, a random access optimized index might be better. The current design does at most one linear pass across each index per round, and likely to a sparse subset based on the relative sizes (driven by galloping).  As long as the number of facts is substantially larger than the number of iterations, this should be .. pretty good, I think?</p>



<a name="128780657"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128780657" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128780657">(Jun 28 2018 at 16:32)</a>:</h4>
<p>It's totally reasonable to ask about swapping in a HashMap or the like, as an actual index, at which point you trade away some throughput and gain some reduction in minimum latency. I have no data on which is better / worse, but one advantage of the sorted lists is that you can re-use a <code>(a,b,c)</code> list for <code>a</code> joins, <code>(a,b)</code> joins, and <code>(a,b,c)</code> semijoins, unlike a <code>HashMap</code> where you need to pick the keys explicitly.</p>



<a name="128780678"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128780678" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128780678">(Jun 28 2018 at 16:33)</a>:</h4>
<p>Factorized stuff could work, but it depends a lot on the query. If there are opportunities for factorization then they can be great, and if there are not then (I think) nothing good results. Nothing stood out at me in the current query. (For Niko's benefit: factorization is maintaining joined results in "cross-product of lists" form rather than expanded out as tuples, most commonly beneficial if another join can be done that can consume the cross product representation (meaning the next keys are not fields left unexpanded).</p>



<a name="128780848"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128780848" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128780848">(Jun 28 2018 at 16:38)</a>:</h4>
<p><span class="user-mention" data-user-id="116609">@Frank McSherry</span> thanks a lot, as always :)</p>



<a name="128780889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128780889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128780889">(Jun 28 2018 at 16:38)</a>:</h4>
<p>I'm not aware of easy idiomatic ways to integrate factorized query stuff, except perhaps hooking the gallop results as an iterator, and allow you to call either <code>flat_map</code> or <code>collect</code>. That being said, I've only come across a query where it helps once (it helped a lot), and I don't have a lot of experience with using them idiomatically.</p>



<a name="128780933"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure/near/128780933" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Frank McSherry <a href="https://rust-lang.github.io/zulip_archive/stream/122657-t-compiler/wg-nll/topic/optimizing-transitive-closure.html#128780933">(Jun 28 2018 at 16:39)</a>:</h4>
<p>Hey no worries! :D</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>